--- title: "5.5.3 XGBoost, LightGBM and CatBoost" description: "Gradient boosting with XGBoost, LightGBM and CatBoost on the same panel: deterministic setup, purged early stopping, sectors and Ridge." url: "https://pythonparatrading.com/en/python-trading-course/level-5/xgboost-lightgbm-catboost-gradient-boosting/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/xgboost-lightgbm-catboost-gradient-boosting.png" --- # 5.5.3 · Gradient boosting: XGBoost, LightGBM and CatBoost > The three gradient boosting libraries are the workhorses of tabular data. This lesson puts them to solve the same problem, with the same configuration and reproducible results, on the level's synthetic panel: it chooses the number of trees in a purged validation, tries the sector as a categorical variable and compares them with the Ridge baseline, which on an almost linear problem they should not beat by much. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.5 · Classical models - Duration: 70 min - Prerequisites: [5.5.1 · Regularized regressions: Ridge, Lasso, Elastic Net and logistic](https://pythonparatrading.com/en/python-trading-course/level-5/ridge-lasso-elastic-net-logistic-regression/), [5.5.2 · Decision trees, Random Forest and bagging](https://pythonparatrading.com/en/python-trading-course/level-5/decision-trees-random-forest-bagging/), [5.4.2 · Purged k-fold with embargo and CPCV for ML models](https://pythonparatrading.com/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv/) ## By the end of this lesson - Explain gradient boosting as a sum of small trees, each fitted to the gradient of the loss of the previous ones, and the role of the learning rate. - Translate a configuration between XGBoost, LightGBM and CatBoost (depth, leaves, subsampling, regularization) and fix everything needed for the result to be deterministic. - Choose the number of trees with a purged validation and the metric that matters (the IC), not with the default early-stopping loss. - Use categorical variables in the three libraries and recognize when a category with no effect overfits. - Compare the boosters with a linear baseline using paired differences and understand why they do not beat it on this panel. ## Boosting: small trees that correct the previous ones A forest (lesson 5.5.2) averages independent trees to lower their variance. **Gradient boosting** does something else: it builds the trees one after another and each one corrects what the previous ones get wrong. With the loss $L(y, F)$ and an initial model $F_0$ (the mean of the label), at each step $m$: 1. the negative gradient of the loss is computed at each row, $r_i = -\partial L(y_i, F_{m-1}(x_i)) / \partial F$ (with squared error, the residual $y_i - F_{m-1}(x_i)$); 2. a small tree $T_m$ is fitted to those $r_i$; 3. it is added with a small weight, the **learning rate** $\eta$: $F_m = F_{m-1} + \eta\,T_m$, and the process is repeated up to $M$ trees. ```mermaid flowchart TB F0["F0 = mean"] -->|residuals| T1[Tree 1] T1 --> F1["F1 = F0 + η·T1"] F1 -->|residuals| T2[Tree 2] T2 --> F2["F2 = F1 + η·T2"] ``` The formulation is Friedman's (2001). XGBoost (Chen and Guestrin, 2016) added a second-order approximation of the loss with L2 regularization on the leaf values and a scalable algorithm; LightGBM (Ke et al., 2017), leaf-wise growth and gradient sampling; CatBoost (Prokhorenkova et al., 2018), symmetric trees and a treatment of categoricals designed to avoid target leakage. All three use histograms: they discretize each feature into a few hundred bins and search for splits only between bins, which makes fitting fast with hundreds of thousands of rows. On tabular data from many domains, boosted trees still beat neural networks on medium-sized benchmarks (Grinsztajn, Oyallon and Varoquaux, 2022). In finance the relevant question is a different one: whether they beat a linear model with good features. On real US equity data, Gu, Kelly and Xiu (2020) find that trees and neural networks outperform penalized regressions and attribute the advantage to **nonlinear interactions between predictors**. If the signal is almost linear, as on this level's panel, there is nothing to gain and variance to pay. This lesson measures it. ## The same problem for all three The data are the level's **synthetic** panel (400 fictitious stocks, 21 weekly features, ground truth published in `public/curso/datos/nivel-5/README.md`; nothing that comes out here says anything about real markets), with features and label in normal ranks within each week as in lesson 5.5.2. Three partitions with a one-week purge between them are needed, because the label of the last week of each stretch ends in the next one (lesson 5.4.2): - **Training** 2012-2016 (259 weeks): to fit with different numbers of trees. - **Validation** 2017-2018 (103 weeks): to choose the number of trees. It is never used to measure the final result. - **Test** 2019-2025 (365 weeks): only once, with the model retrained on 2012-2018. ```python import numpy as np import pandas as pd from scipy.stats import norm from sklearn.linear_model import Ridge f = pd.read_parquet("panel5_features.parquet") e = pd.read_parquet("panel5_etiquetas.parquet") d = (f.merge(e[["fecha", "ticker", "ret_sig_semana"]], on=["fecha", "ticker"]) .dropna(subset=["ret_sig_semana"]) .reset_index(drop=True)) FE = [c for c in f.columns[3:] if c != "revision_analistas"] def g(s): """Normal rank within each week (lesson 5.2.4); missing, 0.""" por = s.groupby(d["fecha"]) u = (por.rank() - 0.5) / por.transform("count") return pd.Series(norm.ppf(u), index=s.index).fillna(0.0) def ic(score, mask): """Rank IC of each week between score and the label.""" t = pd.DataFrame({"s": score, "f": sem[mask], "y": d.loc[mask, "ret_sig_semana"].to_numpy()}) t[["s", "y"]] = t.groupby("f")[["s", "y"]].rank() return t.groupby("f")[["s", "y"]].corr().xs("s", level=1)["y"] X = pd.DataFrame({k: g(d[k]) for k in FE}).astype("float32") X["sector"] = d["sector"].astype("category") yr = g(d["ret_sig_semana"]).to_numpy() sem = d["fecha"].to_numpy() fe = lambda t: np.datetime64(t) ent = sem < fe("2016-12-26") # 2012-2016 val = (sem >= fe("2017-01-06")) & (sem < fe("2018-12-24")) tr_todo = sem < fe("2018-12-24") # 2012-2018 te = sem >= fe("2019-01-01") # 2019-2025 print({k: len(np.unique(sem[m])) for k, m in [("train", ent), ("val", val), ("test", te)]}) ridge = Ridge(alpha=10.0) ic_val_r = ic(ridge.fit(X.loc[ent, FE], yr[ent]) .predict(X.loc[val, FE]), val).mean() ic_r = ic(ridge.fit(X.loc[tr_todo, FE], yr[tr_todo]) .predict(X.loc[te, FE]), te) print(f"Ridge: validation IC {ic_val_r:.4f} test {ic_r.mean():.4f} " f"± {ic_r.std() / np.sqrt(len(ic_r)):.4f}") ``` ```output {'train': 259, 'val': 103, 'test': 365} Ridge: validation IC 0.0155 test 0.0180 ± 0.0045 ``` The baseline: Ridge with α = 10 (with 145,000 rows it is, in practice, OLS; lesson 5.5.1) gives an IC of 0.0155 in the 2017-2018 validation training on 2012-2016, and of 0.0180 ± 0.0045 in the 2019-2025 test training on 2012-2018. The sector is stored as a pandas `category`: that is how XGBoost and LightGBM recognize it as categorical. ## One configuration, three dialects Each library names the same controls differently, and some have no exact equivalent. This is the translation of the lesson's configuration: | Control | XGBoost 3.4 | LightGBM 4.7 | CatBoost 1.2 | Here | |---|---|---|---|---| | Number of trees | `n_estimators` | `n_estimators` | `iterations` | up to 800 | | Learning rate | `learning_rate` | `learning_rate` | `learning_rate` | 0.02 | | Tree size | `max_depth` | `num_leaves` and `max_depth` | `depth` | 2 (4 leaves) | | Minimum rows per leaf | `min_child_weight` (sum of hessians: rows, with squared error) | `min_child_samples` | `min_data_in_leaf` (not with symmetric trees) | 3,000 | | Row subsampling | `subsample` | `subsample` and `subsample_freq` | `subsample` with `bootstrap_type="Bernoulli"` | 0.5 | | Candidate features | `colsample_bytree` | `colsample_bytree` | `rsm` | 0.8 | | Leaf L2 | `reg_lambda` | `reg_lambda` | `l2_leaf_reg` | 10 | | Growth | level-wise (`depthwise`) | leaf-wise (the one with the highest gain) | symmetric trees | | | Categoricals | `enable_categorical=True` and dtype `category` | dtype `category` or `categorical_feature` | `cat_features` | sector | | Determinism | `random_state`, `n_jobs` | `random_state`, `n_jobs`, `deterministic`, `force_row_wise` | `random_seed`, `thread_count` | 0 and 4 threads | Three fundamental differences. **Growth**: XGBoost splits all the nodes of a level before moving on to the next one; LightGBM splits the leaf with the largest loss reduction, wherever it is, and that is why its size is controlled with `num_leaves` (with `max_depth` as a brake); CatBoost uses the **same split in all the nodes of a level** (symmetric or *oblivious* trees), heavily regularized and fast to evaluate. With depth 2 the three trees have at most four leaves, but they are not the same: CatBoost's uses a single question per level. **Minimum leaf**: CatBoost does not support it with symmetric trees; its regularization comes from the symmetry and from `l2_leaf_reg`. **Categoricals**: each handles them in its own way (section “The sector as a categorical variable”). Why depth 2, 3,000-row leaves and a rate of 0.02? Because of what lesson 5.5.2 taught: on this panel, the useful trees are tiny and average many rows. A booster with the default values (depth 6 in XGBoost and CatBoost, 31 leaves in LightGBM, rate 0.1 to 0.3) overfits in a few iterations. ```python import catboost as cb import lightgbm as lgb import xgboost as xgb N, LR = 800, 0.02 def modelos(semilla=0): return { "xgboost": xgb.XGBRegressor( n_estimators=N, learning_rate=LR, max_depth=2, min_child_weight=3000, subsample=0.5, colsample_bytree=0.8, reg_lambda=10, tree_method="hist", enable_categorical=True, random_state=semilla, n_jobs=4), "lightgbm": lgb.LGBMRegressor( n_estimators=N, learning_rate=LR, max_depth=2, num_leaves=4, min_child_samples=3000, subsample=0.5, subsample_freq=1, colsample_bytree=0.8, reg_lambda=10, random_state=semilla, n_jobs=4, deterministic=True, force_row_wise=True, verbose=-1), "catboost": cb.CatBoostRegressor( iterations=N, learning_rate=LR, depth=2, l2_leaf_reg=10, subsample=0.5, bootstrap_type="Bernoulli", rsm=0.8, random_seed=semilla, thread_count=4, verbose=0, allow_writing_files=False), } def predice(nombre, m, Xm, k): """Prediction with the first k trees.""" if nombre == "xgboost": return m.predict(Xm, iteration_range=(0, k)) if nombre == "lightgbm": return m.predict(Xm, num_iteration=k) return m.predict(Xm, ntree_end=k) K = [25, 50, 100, 200, 400, 800] ajustados, curva = {}, {} for nombre, m in modelos().items(): ajustados[nombre] = m.fit(X.loc[ent, FE], yr[ent]) curva[nombre] = [ic(predice(nombre, m, X.loc[val, FE], k), val).mean() for k in K] pd.DataFrame(curva, index=K).round(4) ``` ```output xgboost lightgbm catboost 25 0.0098 0.0077 0.0058 50 0.0089 0.0073 0.0049 100 0.0083 0.0075 0.0069 200 0.0077 0.0070 0.0088 400 0.0055 0.0063 0.0095 800 0.0035 0.0041 0.0086 ``` `verbose=-1` in LightGBM, and `verbose=0` and `allow_writing_files=False` in CatBoost (otherwise it creates a `catboost_info/` folder with the training logs) keep the output clean. Each library has its own way of predicting with the first $k$ trees of an already trained model (`iteration_range`, `num_iteration`, `ntree_end`): that is how the whole curve is obtained with a single fit. *Fig. — Validation IC (2017-2018, purged) of the three boosters by number of trees, training on 2012-2016, against Ridge.* Three curves on a log scale from 25 to 800 trees. XGBoost (dashed) starts at 0.0098 with 25 trees and falls to 0.0035 with 800; LightGBM falls from 0.0077 to 0.0041; CatBoost rises from 0.0058 to a maximum of 0.0095 at 400 trees and falls to 0.0086. All three stay below the Ridge line (highlighted), 0.0155. The figure draws the table. XGBoost and LightGBM give their best validation IC with **25 trees**, the lower edge of the grid (the optimum could be even lower), and get worse with every tree added: from 0.0098 to 0.0035 and from 0.0077 to 0.0041. CatBoost, with stiffer symmetric trees, rises up to 400 trees (0.0095) and drops a little after that. All three stay below Ridge (0.0155) over the whole range. Unlike the forest of lesson 5.5.2, boosting **does overfit as trees are added**: each one is fitted to the residuals of the previous ones, and with this label the residuals are almost all noise. ## Early stopping looks at the loss, not the IC All three libraries have early stopping: they are given a validation set, evaluate the loss after each tree and stop when it has gone $n$ iterations without improving. It is convenient, and you need to know what it measures. ```python lg = lgb.LGBMRegressor(**modelos()["lightgbm"].get_params()) lg.fit(X.loc[ent, FE], yr[ent], eval_X=(X.loc[val, FE],), eval_y=(yr[val],), callbacks=[lgb.early_stopping(100, verbose=False)]) print("best iteration by squared error:", lg.best_iteration_) ``` ```output best iteration by squared error: 14 ``` With the validation squared error, LightGBM stops at 14 trees. That is not the same as the IC maximum: the squared error punishes the **scale** of the predictions, and with such a weak signal any prediction with variance makes it worse even if it ranks somewhat better; a decile portfolio only uses the **ordering**. That is why the previous cell computes the validation IC, week by week, for each number of trees. You could pass a custom metric to early stopping (an `eval_metric` that groups by week), but the explicit curve also shows how flat and noisy it is. Two version details. In LightGBM 4.7, `eval_set=[(X, y)]` emits a deprecation warning: the validation sets go in `eval_X` and `eval_y`. And the validation has to be **purged**: if the labels were 21-session ones, a validation right next to the training set would share four weeks of label with it and early stopping would choose a model that memorizes (lessons 5.4.1 and 5.4.2). ## The chosen number of trees and the test With the $k$ of highest validation IC, each model is **retrained on all of 2012-2018** and evaluated only once on 2019-2025. The difference with Ridge is measured paired, week by week. ```python filas = [] for nombre in ajustados: k = K[int(np.argmax(curva[nombre]))] m = modelos()[nombre] m.set_params(**{"iterations" if nombre == "catboost" else "n_estimators": k}) m.fit(X.loc[tr_todo, FE], yr[tr_todo]) s = ic(m.predict(X.loc[te, FE]), te) dif = s - ic_r filas.append({"modelo": nombre, "k": k, "ic_val": max(curva[nombre]), "ic_prueba": s.mean(), "se": s.std() / np.sqrt(len(s)), "menos_ridge": dif.mean(), "se_dif": dif.std() / np.sqrt(len(dif))}) pd.DataFrame(filas).set_index("modelo").round(4) ``` ```output k ic_val ic_prueba se menos_ridge se_dif modelo xgboost 25 0.0098 0.0136 0.0046 -0.0044 0.0022 lightgbm 25 0.0077 0.0139 0.0047 -0.0041 0.0022 catboost 400 0.0095 0.0160 0.0046 -0.0020 0.0019 ``` The figure at the top summarizes the result. XGBoost gives 0.0136, LightGBM 0.0139 and CatBoost 0.0160, versus 0.0180 for Ridge and 0.0256 for the oracle over the same period (lesson 5.5.1). The paired differences with Ridge are negative for all three: −0.0044 ± 0.0022 and −0.0041 ± 0.0022 (t ≈ −2) for XGBoost and LightGBM, −0.0020 ± 0.0019 for CatBoost. It is the expected result on this panel: the ground truth is almost linear in the ranks, the only planted interaction is small (lesson 5.5.2) and the boosters pay for it with variance. With the libraries' default values the result would be worse. ## The sector as a categorical variable The sector is the typical categorical variable of a stock panel. On the panel it **has no planted effect**: the true alpha does not depend on it (the sectors have different betas, but that cannot be predicted without knowing where the market will go). The right outcome would be for it not to help. The three libraries treat it differently: - **LightGBM** sorts the categories of each node by their mean gradient (`sum_gradient / sum_hessian`, according to its documentation) and searches for the best partition into two groups along that ordering. With ten sectors it is a very flexible split. - **XGBoost** with `enable_categorical=True` does something similar: with fewer categories than `max_cat_to_onehot` (4 by default) it tries one against the rest; with more, it distributes the categories between the node's two children. - **CatBoost** replaces each category with **target statistics** computed in order: it goes through the rows in a permutation and encodes each one only with the labels of the previous ones. That way the encoding of a row does not include its own label, a target leakage that its authors identify and correct. ```python COL = FE + ["sector"] k = 200 cat = { "xgboost": xgb.XGBRegressor(**{**modelos()["xgboost"] .get_params(), "n_estimators": k}), "lightgbm": lgb.LGBMRegressor(**{**modelos()["lightgbm"] .get_params(), "n_estimators": k}), "catboost": cb.CatBoostRegressor( **{**modelos()["catboost"].get_params(), "iterations": k}, cat_features=["sector"]), } filas = [] for nombre, m in cat.items(): m.fit(X.loc[ent, COL], yr[ent]) sin = ic(predice(nombre, ajustados[nombre], X.loc[val, FE], k), val).mean() con = ic(m.predict(X.loc[val, COL]), val).mean() filas.append({"modelo": nombre, "ic_val_sin_sector": sin, "ic_val_con_sector": con}) pd.DataFrame(filas).set_index("modelo").round(4) ``` ```output ic_val_sin_sector ic_val_con_sector modelo xgboost 0.0077 0.0043 lightgbm 0.0070 0.0039 catboost 0.0088 0.0077 ``` *Fig. — Validation IC of each booster without the sector and with the sector as a native categorical variable (or one-hot), with 200 trees: the sector has no planted effect.* Four rows. XGBoost falls from 0.0077 without sector to 0.0043 with sector; LightGBM with the native categorical (highlighted), from 0.0070 to 0.0039; CatBoost, with its ordered target statistics, from 0.0088 to 0.0077; LightGBM with the sector as ten one-hot columns, from 0.0070 to 0.0068. Adding a variable with no effect hurts all of them; less the more the method guards against overfitting categories. With 200 trees, adding the sector lowers XGBoost's validation IC from 0.0077 to 0.0043 and LightGBM's from 0.0070 to 0.0039: almost by half. CatBoost also gets worse, but less (from 0.0088 to 0.0077). The figure adds the result of exercise 2: the same LightGBM with the sector in ten *one-hot* columns barely changes (0.0068). The next cell shows why. ```python cortes = pd.Series(cat["lightgbm"].booster_.feature_importance("split"), index=COL) print(f"splits on sector: {cortes['sector']} of {cortes.sum()}") cortes.sort_values(ascending=False).head(6) ``` ```output splits on sector: 104 of 547 sector 104 ret_1s 77 ruido_5 35 mom_12_1 34 sue 31 ret_1m 27 dtype: int32 ``` The sector is the feature with the most splits in the model, 104 of 547, ahead of `ret_1s` (77), the strongest effect in the panel. A partition of ten categories chosen with the objective itself almost always finds something that improves the training fit: with 511 ways of splitting ten sectors into two groups, there is always one that matches the noise of those weeks. Behind it appears `ruido_5`, fat-tailed noise, with 35 splits. A categorical with many levels and no effect is a source of overfitting; if it has to be used, it is better with CatBoost's protection, as one-hot or as a statistic computed outside the training sample. ## Determinism and speed The course requires every output to be identical on every run. The three libraries allow it on CPU with the settings in the table; the cell checks it by fitting each model twice. ```python for nombre in ["xgboost", "lightgbm", "catboost"]: a = modelos()[nombre] a.set_params(**{"iterations" if nombre == "catboost" else "n_estimators": 100}) p1 = a.fit(X.loc[ent, FE], yr[ent]).predict(X.loc[val, FE]) p2 = a.fit(X.loc[ent, FE], yr[ent]).predict(X.loc[val, FE]) print(f"{nombre:9s} identical: {np.array_equal(p1, p2)}") ``` ```output xgboost identical: True lightgbm identical: True catboost identical: True ``` Identical does not mean the same on another machine: with another CPU or another number of threads, floating-point sums can change in the last decimals and, on a split that sits right at the limit, change a tree. That is why the course outputs are rounded to what the conclusion needs, and in production the trained model is stored instead of retraining it to reproduce a prediction. On GPU (`device="cuda"` in XGBoost, `task_type="GPU"` in CatBoost) there are operations that do not guarantee the same summation order; the course only uses CPU. Speed is easy to measure and hard to compare, because it depends on the machine, the threads and the load. The code to measure it, not run in the course (its timings would change on every run): ```python skip import time for nombre, m in modelos().items(): t0 = time.perf_counter() m.fit(X.loc[tr_todo, FE], yr[tr_todo]) print(f"{nombre:9s} {time.perf_counter() - t0:5.1f} s") ``` On the machine where the lesson was written (16 cores shared with other processes, 4 threads per model), 800 trees with 145,200 rows took about 3 s in XGBoost, 4 s in LightGBM and 4-4.5 s in CatBoost; Ridge, 0.04 s. With `deterministic=True` and `force_row_wise=True` LightGBM gives up part of its speed, and with depth-2 trees the advantage of leaf-wise growth does not show. Published speed comparisons use other data and other settings: the only sure thing is that all three are fast enough to retrain every week and that Ridge is a hundred times faster. ## Common mistakes - **Using the default values.** Depth 6 or 31 leaves and a rate of 0.1-0.3 are designed for problems with signal; on returns they overfit within a few trees. - **Stopping on the loss instead of the objective's metric.** The squared error chose 14 trees; the IC, 25. And in any case, purged validation. - **Choosing the number of trees on the test.** The 2019-2025 curve is not looked at to decide; that is what the validation is for. After choosing, retrain with training and validation together. - **Adding categoricals with no effect.** The sector takes more splits than the best feature and halves the IC. - **Forgetting the baseline.** An IC of 0.0160 looks good until you see that Ridge, over the same weeks and in a fraction of the time, gives 0.0180. - **Half-way reproducibility.** Without `deterministic=True` and `force_row_wise=True` in LightGBM, or without `allow_writing_files=False` in CatBoost, the result or the working directory changes on every run. ## Exercises 1. The comparison with Ridge was made with a single partition. Repeat it with an annual walk-forward from 2015 to 2025 (retraining every January on everything before it, with a one-week purge) for LightGBM with 25 trees. Does it hold? **Solution.** ```python anio = pd.DatetimeIndex(sem).year.to_numpy() p_lg = np.full(len(d), np.nan) p_r = np.full(len(d), np.nan) for a in range(2015, 2026): tr = sem < fe(f"{a}-01-01") - np.timedelta64(7, "D") t_ = anio == a m = modelos()["lightgbm"].set_params(n_estimators=25) p_lg[t_] = m.fit(X.loc[tr, FE], yr[tr]).predict(X.loc[t_, FE]) p_r[t_] = (Ridge(alpha=10.0).fit(X.loc[tr, FE], yr[tr]) .predict(X.loc[t_, FE])) wf = anio >= 2015 i_lg, i_r = ic(p_lg[wf], wf), ic(p_r[wf], wf) dif = i_lg - i_r print(f"LightGBM {i_lg.mean():.4f} Ridge {i_r.mean():.4f} " f"difference {dif.mean():+.4f} ± " f"{dif.std() / np.sqrt(len(dif)):.4f}") ``` ```output LightGBM 0.0137 Ridge 0.0178 difference -0.0041 ± 0.0018 ``` It holds: over 574 test weeks, LightGBM gives 0.0137 and Ridge 0.0178, a difference of −0.0041 ± 0.0018 (t ≈ −2.3). The number of trees (25) was chosen with the 2017-2018 validation, which is part of several training sets of this walk-forward; a completely clean procedure would choose it inside each year, as was done with the α of lesson 5.5.1, and that could only make the booster worse. 2. Encode the sector in ten *one-hot* columns (`pd.get_dummies`) and repeat the LightGBM measurement with 200 trees. Why does one-hot do less damage than the native categorical? **Solution.** ```python uno = pd.get_dummies(d["sector"], prefix="s", dtype="float32") Xo = pd.concat([X[FE], uno], axis=1) m = modelos()["lightgbm"].set_params(n_estimators=200) m.fit(Xo[ent], yr[ent]) print(f"validation IC with sector as one-hot: " f"{ic(m.predict(Xo[val]), val).mean():.4f}") ``` ```output validation IC with sector as one-hot: 0.0068 ``` The validation IC stays at 0.0068, versus 0.0070 without the sector and 0.0039 with the native categorical. With one-hot, each split can only separate one sector from the rest (ten options instead of 511 partitions), and a column with 10% ones rarely beats a continuous feature with leaves of at least 3,000 rows. The LightGBM documentation says that one-hot is suboptimal for trees, and it is when the category has an effect: several splits are needed to separate groups of categories. When it has none, that limitation is exactly what protects. 3. No code: a colleague replaces the production Ridge with a LightGBM because “in the backtest the IC rises from 0.018 to 0.031”. The LightGBM has 31 leaves, 1,000 trees, a rate of 0.1, the sector and the industry as categoricals, and the number of trees was chosen with early stopping on the last two years of the backtest. What would you check? **Solution.** First, the validation: if the last two years of the backtest were used for early stopping, the model was chosen with the data it is evaluated on; a test that no decision has touched is needed, with purging and an embargo (lessons 5.4.2 and 5.4.3). Second, the configuration: 31 leaves, a rate of 0.1 and 1,000 trees overfit on returns within very few iterations, and an IC of 0.031 is more than twice that of a good linear model; either the model found interactions that exist (you must be able to name them and see them in a partial dependence, lesson 5.5.2) or there is a leak. Third, the categoricals: sector and industry with native encoding take splits with no real effect; I would compare the result without them and with one-hot. Finally, the paired difference with Ridge week by week, the number of configurations tried to correct with the Deflated Sharpe (lesson 5.4.4) and the backtest with costs, because a more nervous model usually trades more. ## Self-assessment 1. With a learning rate of 0.02, depth-2 trees and a purged 2017-2018 validation, the validation IC of XGBoost is 0.0098 with 25 trees and 0.0035 with 800. What does it indicate? - A. That with more trees the model learns the signal little by little - B. That the validation has a leak - C. That the learning rate has to be raised - D. That each new tree mostly fits training noise: with this signal-to-noise ratio, boosting overfits right away and the number of trees is a hyperparameter that has to be validated *Answer D: That each new tree mostly fits training noise: with this signal-to-noise ratio, boosting overfits right away and the number of trees is a hyperparameter that has to be validated.* Unlike a forest, boosting does overfit when trees are added: each one corrects the residuals of the previous ones, and the residuals of such a label are almost all noise. The maximum of XGBoost and LightGBM is at the lower edge of the grid (25 trees). 2. LightGBM early stopping with the validation squared error stops at iteration 14. Why is it not a good default choice here? - A. Because 14 trees are too few for any problem - B. Because LightGBM does not allow early stopping with validation - C. Because the squared error penalizes the scale of the predictions and what is traded is the ordering: the stopping metric must be that of the objective (the weekly IC), computed by week - D. Because the squared error always chooses too many trees *Answer C: Because the squared error penalizes the scale of the predictions and what is traded is the ordering: the stopping metric must be that of the objective (the weekly IC), computed by week.* With such a weak signal, almost any prediction with variance increases the squared error even if it ranks the stocks somewhat better. A long-short decile portfolio only uses the ordering. That is why the lesson computes the validation IC for each number of trees. 3. Adding the sector (with no planted effect) as a native categorical lowers the LightGBM validation IC from 0.0070 to 0.0039 and the sector is the feature with the most splits (104 of 547). What mechanism explains it? - A. The sector has an effect that the generator does not document - B. LightGBM sorts the categories by their mean gradient in each node and splits the set into two groups: with ten sectors there are many possible partitions and one of them always fits training noise - C. The sector is wrongly encoded - D. Categoricals cannot be used with trees *Answer B: LightGBM sorts the categories by their mean gradient in each node and splits the set into two groups: with ten sectors there are many possible partitions and one of them always fits training noise.* Choosing the best partition of ten categories according to the objective itself is a very flexible fit. CatBoost protects with target statistics computed in order (it only drops from 0.0088 to 0.0077) and one-hot limits it to splits of one category against the rest (0.0070 to 0.0068). 4. In the 2019-2025 test, the three boosters give ICs from 0.0136 to 0.0160 versus 0.0180 for Ridge. Which conclusion is correct? - A. That on this panel the signal is almost linear in the ranks and the boosters pay in variance what they do not gain in flexibility; on data with interactions that matter they can win, and that is why they are always compared with the linear baseline - B. That gradient boosting is useless for finance - C. That more trees have to be used - D. That CatBoost is better than XGBoost on any problem *Answer A: That on this panel the signal is almost linear in the ranks and the boosters pay in variance what they do not gain in flexibility; on data with interactions that matter they can win, and that is why they are always compared with the linear baseline.* 61% of the panel's ground truth is explained by a model that is linear in the ranks (lesson 5.5.1). On real US data, Gu, Kelly and Xiu (2020) attribute the advantage of trees and neural networks to nonlinear interactions between predictors; if they do not exist or are not found, flexibility only adds variance. 5. What has to be fixed in LightGBM for two fits with the same data to give exactly the same prediction? - A. Only `random_state` - B. Nothing: LightGBM is always deterministic - C. `random_state`, a fixed `n_jobs`, `deterministic=True` and `force_row_wise=True` - D. A single thread has to be used *Answer C: `random_state`, a fixed `n_jobs`, `deterministic=True` and `force_row_wise=True`.* With subsampling the seed is needed; with several threads, `deterministic=True` and a fixed histogram construction strategy (`force_row_wise`). In XGBoost `random_state` with `tree_method="hist"` on CPU is enough; in CatBoost, `random_seed` and `thread_count`. ## Key points - Gradient boosting sums small trees; each one is fitted to the gradient of the loss of the previous ones and enters multiplied by the learning rate. Unlike a forest, it overfits as trees are added. - With the same configuration (depth 2, 3,000-row leaves, subsampling 0.5, L2 = 10, rate 0.02), XGBoost and LightGBM reach their best validation IC with 25 trees and CatBoost with 400. None reaches that of Ridge (0.0155). - Out-of-the-box early stopping looks at the loss (squared error: 14 trees), not at the IC. The number of trees is chosen with the objective's metric, in a purged validation, and the model is retrained on the whole training set. - Out of sample, all three fall below Ridge (paired differences of −0.0044, −0.0041 and −0.0020); in an annual walk-forward, LightGBM loses −0.0041 ± 0.0018. On an almost linear problem, flexibility costs variance. - A categorical with no effect makes all three worse: native LightGBM goes from 0.0070 to 0.0039 validation IC; CatBoost, with ordered statistics, only from 0.0088 to 0.0077; one-hot barely changes anything. - Determinism: seeds, fixed threads and, in LightGBM, `deterministic=True` and `force_row_wise=True`. The three libraries give identical predictions in two fits. ## Resources ### Documentation - [XGBoost: official documentation](https://xgboost.readthedocs.io/en/stable/) · xgboost.readthedocs.io - [LightGBM: official documentation (features: leaf-wise growth and categoricals)](https://lightgbm.readthedocs.io/en/stable/Features.html) · lightgbm.readthedocs.io - [CatBoost: official documentation](https://catboost.ai/docs/en/) · catboost.ai ### Books - [Friedman, J. H. (2001): Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics 29(5), 1189-1232](https://doi.org/10.1214/aos/1013203451) · doi.org - [Chen, T. and Guestrin, C. (2016): XGBoost: A Scalable Tree Boosting System. KDD '16, 785-794](https://doi.org/10.1145/2939672.2939785) · doi.org - [Ke, G. et al. (2017): LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS 30](https://papers.nips.cc/paper_files/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html) · papers.nips.cc - [Prokhorenkova, L. et al. (2018): CatBoost: unbiased boosting with categorical features (arXiv:1706.09516)](https://arxiv.org/abs/1706.09516) · arxiv.org - [Grinsztajn, L., Oyallon, E. and Varoquaux, G. (2022): Why do tree-based models still outperform deep learning on tabular data? (arXiv:2207.08815)](https://arxiv.org/abs/2207.08815) · arxiv.org Previous lesson: [5.5.2 · Decision trees, Random Forest and bagging](https://pythonparatrading.com/en/python-trading-course/level-5/decision-trees-random-forest-bagging/) Next lesson: [5.5.4 · Optuna and the risk of overfitting the validation](https://pythonparatrading.com/en/python-trading-course/level-5/optuna-validation-overfitting/)