Gradient boosting: XGBoost, LightGBM and CatBoost.

Boosting: small trees that correct the previous ones

A forest (lesson 5.5.2) averages independent trees to lower their variance. Gradient boosting does something else: it builds the trees one after another and each one corrects what the previous ones get wrong. With the loss $L(y, F)$ and an initial model $F_0$ (the mean of the label), at each step $m$:

1. the negative gradient of the loss is computed at each row, $r_i = -\partial L(y_i, F_{m-1}(x_i)) / \partial F$ (with squared error, the residual $y_i - F_{m-1}(x_i)$);
2. a small tree $T_m$ is fitted to those $r_i$;
3. it is added with a small weight, the learning rate $\eta$: $F_m = F_{m-1} + \eta\,T_m$, and the process is repeated up to $M$ trees.

flowchart TB
  F0["F0 = mean"] -->|residuals| T1[Tree 1]
  T1 --> F1["F1 = F0 + η·T1"]
  F1 -->|residuals| T2[Tree 2]
  T2 --> F2["F2 = F1 + η·T2"]

The formulation is Friedman's (2001). XGBoost (Chen and Guestrin, 2016) added a second-order approximation of the loss with L2 regularization on the leaf values and a scalable algorithm; LightGBM (Ke et al., 2017), leaf-wise growth and gradient sampling; CatBoost (Prokhorenkova et al., 2018), symmetric trees and a treatment of categoricals designed to avoid target leakage. All three use histograms: they discretize each feature into a few hundred bins and search for splits only between bins, which makes fitting fast with hundreds of thousands of rows.

On tabular data from many domains, boosted trees still beat neural networks on medium-sized benchmarks (Grinsztajn, Oyallon and Varoquaux, 2022). In finance the relevant question is a different one: whether they beat a linear model with good features. On real US equity data, Gu, Kelly and Xiu (2020) find that trees and neural networks outperform penalized regressions and attribute the advantage to nonlinear interactions between predictors. If the signal is almost linear, as on this level's panel, there is nothing to gain and variance to pay. This lesson measures it.

The same problem for all three

The data are the level's synthetic panel (400 fictitious stocks, 21 weekly features, ground truth published in public/curso/datos/nivel-5/README.md; nothing that comes out here says anything about real markets), with features and label in normal ranks within each week as in lesson 5.5.2. Three partitions with a one-week purge between them are needed, because the label of the last week of each stretch ends in the next one (lesson 5.4.2):

• Training 2012-2016 (259 weeks): to fit with different numbers of trees.
• Validation 2017-2018 (103 weeks): to choose the number of trees. It is never used to measure the final result.
• Test 2019-2025 (365 weeks): only once, with the model retrained on 2012-2018.

import numpy as np
import pandas as pd
from scipy.stats import norm
from sklearn.linear_model import Ridge

f = pd.read_parquet("panel5_features.parquet")
e = pd.read_parquet("panel5_etiquetas.parquet")
d = (f.merge(e[["fecha", "ticker", "ret_sig_semana"]],
             on=["fecha", "ticker"])
      .dropna(subset=["ret_sig_semana"])
      .reset_index(drop=True))
FE = [c for c in f.columns[3:] if c != "revision_analistas"]

def g(s):
    """Normal rank within each week (lesson 5.2.4); missing, 0."""
    por = s.groupby(d["fecha"])
    u = (por.rank() - 0.5) / por.transform("count")
    return pd.Series(norm.ppf(u), index=s.index).fillna(0.0)

def ic(score, mask):
    """Rank IC of each week between score and the label."""
    t = pd.DataFrame({"s": score, "f": sem[mask],
                      "y": d.loc[mask, "ret_sig_semana"].to_numpy()})
    t[["s", "y"]] = t.groupby("f")[["s", "y"]].rank()
    return t.groupby("f")[["s", "y"]].corr().xs("s", level=1)["y"]

X = pd.DataFrame({k: g(d[k]) for k in FE}).astype("float32")
X["sector"] = d["sector"].astype("category")
yr = g(d["ret_sig_semana"]).to_numpy()
sem = d["fecha"].to_numpy()
fe = lambda t: np.datetime64(t)
ent = sem < fe("2016-12-26")                          # 2012-2016
val = (sem >= fe("2017-01-06")) & (sem < fe("2018-12-24"))
tr_todo = sem < fe("2018-12-24")                      # 2012-2018
te = sem >= fe("2019-01-01")                          # 2019-2025
print({k: len(np.unique(sem[m])) for k, m in
       [("train", ent), ("val", val), ("test", te)]})

ridge = Ridge(alpha=10.0)
ic_val_r = ic(ridge.fit(X.loc[ent, FE], yr[ent])
              .predict(X.loc[val, FE]), val).mean()
ic_r = ic(ridge.fit(X.loc[tr_todo, FE], yr[tr_todo])
          .predict(X.loc[te, FE]), te)
print(f"Ridge: validation IC {ic_val_r:.4f}  test {ic_r.mean():.4f} "
      f"± {ic_r.std() / np.sqrt(len(ic_r)):.4f}")

{'train': 259, 'val': 103, 'test': 365}
Ridge: validation IC 0.0155  test 0.0180 ± 0.0045

The baseline: Ridge with α = 10 (with 145,000 rows it is, in practice, OLS; lesson 5.5.1) gives an IC of 0.0155 in the 2017-2018 validation training on 2012-2016, and of 0.0180 ± 0.0045 in the 2019-2025 test training on 2012-2018. The sector is stored as a pandas category: that is how XGBoost and LightGBM recognize it as categorical.

One configuration, three dialects

Each library names the same controls differently, and some have no exact equivalent. This is the translation of the lesson's configuration:

Three fundamental differences. Growth: XGBoost splits all the nodes of a level before moving on to the next one; LightGBM splits the leaf with the largest loss reduction, wherever it is, and that is why its size is controlled with num_leaves (with max_depth as a brake); CatBoost uses the same split in all the nodes of a level (symmetric or oblivious trees), heavily regularized and fast to evaluate. With depth 2 the three trees have at most four leaves, but they are not the same: CatBoost's uses a single question per level. Minimum leaf: CatBoost does not support it with symmetric trees; its regularization comes from the symmetry and from l2_leaf_reg. Categoricals: each handles them in its own way (section “The sector as a categorical variable”).

Why depth 2, 3,000-row leaves and a rate of 0.02? Because of what lesson 5.5.2 taught: on this panel, the useful trees are tiny and average many rows. A booster with the default values (depth 6 in XGBoost and CatBoost, 31 leaves in LightGBM, rate 0.1 to 0.3) overfits in a few iterations.

import catboost as cb
import lightgbm as lgb
import xgboost as xgb

N, LR = 800, 0.02
def modelos(semilla=0):
    return {
        "xgboost": xgb.XGBRegressor(
            n_estimators=N, learning_rate=LR, max_depth=2,
            min_child_weight=3000, subsample=0.5, colsample_bytree=0.8,
            reg_lambda=10, tree_method="hist", enable_categorical=True,
            random_state=semilla, n_jobs=4),
        "lightgbm": lgb.LGBMRegressor(
            n_estimators=N, learning_rate=LR, max_depth=2, num_leaves=4,
            min_child_samples=3000, subsample=0.5, subsample_freq=1,
            colsample_bytree=0.8, reg_lambda=10, random_state=semilla,
            n_jobs=4, deterministic=True, force_row_wise=True,
            verbose=-1),
        "catboost": cb.CatBoostRegressor(
            iterations=N, learning_rate=LR, depth=2, l2_leaf_reg=10,
            subsample=0.5, bootstrap_type="Bernoulli", rsm=0.8,
            random_seed=semilla, thread_count=4, verbose=0,
            allow_writing_files=False),
    }

def predice(nombre, m, Xm, k):
    """Prediction with the first k trees."""
    if nombre == "xgboost":
        return m.predict(Xm, iteration_range=(0, k))
    if nombre == "lightgbm":
        return m.predict(Xm, num_iteration=k)
    return m.predict(Xm, ntree_end=k)

K = [25, 50, 100, 200, 400, 800]
ajustados, curva = {}, {}
for nombre, m in modelos().items():
    ajustados[nombre] = m.fit(X.loc[ent, FE], yr[ent])
    curva[nombre] = [ic(predice(nombre, m, X.loc[val, FE], k),
                        val).mean() for k in K]
pd.DataFrame(curva, index=K).round(4)

     xgboost  lightgbm  catboost
25    0.0098    0.0077    0.0058
50    0.0089    0.0073    0.0049
100   0.0083    0.0075    0.0069
200   0.0077    0.0070    0.0088
400   0.0055    0.0063    0.0095
800   0.0035    0.0041    0.0086

verbose=-1 in LightGBM, and verbose=0 and allow_writing_files=False in CatBoost (otherwise it creates a catboost_info/ folder with the training logs) keep the output clean. Each library has its own way of predicting with the first $k$ trees of an already trained model (iteration_range, num_iteration, ntree_end): that is how the whole curve is obtained with a single fit.

The figure draws the table. XGBoost and LightGBM give their best validation IC with 25 trees, the lower edge of the grid (the optimum could be even lower), and get worse with every tree added: from 0.0098 to 0.0035 and from 0.0077 to 0.0041. CatBoost, with stiffer symmetric trees, rises up to 400 trees (0.0095) and drops a little after that. All three stay below Ridge (0.0155) over the whole range. Unlike the forest of lesson 5.5.2, boosting does overfit as trees are added: each one is fitted to the residuals of the previous ones, and with this label the residuals are almost all noise.

Early stopping looks at the loss, not the IC

All three libraries have early stopping: they are given a validation set, evaluate the loss after each tree and stop when it has gone $n$ iterations without improving. It is convenient, and you need to know what it measures.

lg = lgb.LGBMRegressor(**modelos()["lightgbm"].get_params())
lg.fit(X.loc[ent, FE], yr[ent], eval_X=(X.loc[val, FE],),
       eval_y=(yr[val],),
       callbacks=[lgb.early_stopping(100, verbose=False)])
print("best iteration by squared error:", lg.best_iteration_)

best iteration by squared error: 14

With the validation squared error, LightGBM stops at 14 trees. That is not the same as the IC maximum: the squared error punishes the scale of the predictions, and with such a weak signal any prediction with variance makes it worse even if it ranks somewhat better; a decile portfolio only uses the ordering. That is why the previous cell computes the validation IC, week by week, for each number of trees. You could pass a custom metric to early stopping (an eval_metric that groups by week), but the explicit curve also shows how flat and noisy it is.

Two version details. In LightGBM 4.7, eval_set=[(X, y)] emits a deprecation warning: the validation sets go in eval_X and eval_y. And the validation has to be purged: if the labels were 21-session ones, a validation right next to the training set would share four weeks of label with it and early stopping would choose a model that memorizes (lessons 5.4.1 and 5.4.2).

The chosen number of trees and the test

With the $k$ of highest validation IC, each model is retrained on all of 2012-2018 and evaluated only once on 2019-2025. The difference with Ridge is measured paired, week by week.

filas = []
for nombre in ajustados:
    k = K[int(np.argmax(curva[nombre]))]
    m = modelos()[nombre]
    m.set_params(**{"iterations" if nombre == "catboost"
                    else "n_estimators": k})
    m.fit(X.loc[tr_todo, FE], yr[tr_todo])
    s = ic(m.predict(X.loc[te, FE]), te)
    dif = s - ic_r
    filas.append({"modelo": nombre, "k": k,
                  "ic_val": max(curva[nombre]), "ic_prueba": s.mean(),
                  "se": s.std() / np.sqrt(len(s)),
                  "menos_ridge": dif.mean(),
                  "se_dif": dif.std() / np.sqrt(len(dif))})
pd.DataFrame(filas).set_index("modelo").round(4)

            k  ic_val  ic_prueba      se  menos_ridge  se_dif
modelo
xgboost    25  0.0098     0.0136  0.0046      -0.0044  0.0022
lightgbm   25  0.0077     0.0139  0.0047      -0.0041  0.0022
catboost  400  0.0095     0.0160  0.0046      -0.0020  0.0019

The figure at the top summarizes the result. XGBoost gives 0.0136, LightGBM 0.0139 and CatBoost 0.0160, versus 0.0180 for Ridge and 0.0256 for the oracle over the same period (lesson 5.5.1). The paired differences with Ridge are negative for all three: −0.0044 ± 0.0022 and −0.0041 ± 0.0022 (t ≈ −2) for XGBoost and LightGBM, −0.0020 ± 0.0019 for CatBoost. It is the expected result on this panel: the ground truth is almost linear in the ranks, the only planted interaction is small (lesson 5.5.2) and the boosters pay for it with variance. With the libraries' default values the result would be worse.

The sector as a categorical variable

The sector is the typical categorical variable of a stock panel. On the panel it has no planted effect: the true alpha does not depend on it (the sectors have different betas, but that cannot be predicted without knowing where the market will go). The right outcome would be for it not to help. The three libraries treat it differently:

• LightGBM sorts the categories of each node by their mean gradient (sum_gradient / sum_hessian, according to its documentation) and searches for the best partition into two groups along that ordering. With ten sectors it is a very flexible split.
• XGBoost with enable_categorical=True does something similar: with fewer categories than max_cat_to_onehot (4 by default) it tries one against the rest; with more, it distributes the categories between the node's two children.
• CatBoost replaces each category with target statistics computed in order: it goes through the rows in a permutation and encodes each one only with the labels of the previous ones. That way the encoding of a row does not include its own label, a target leakage that its authors identify and correct.

COL = FE + ["sector"]
k = 200
cat = {
    "xgboost": xgb.XGBRegressor(**{**modelos()["xgboost"]
                                   .get_params(), "n_estimators": k}),
    "lightgbm": lgb.LGBMRegressor(**{**modelos()["lightgbm"]
                                     .get_params(), "n_estimators": k}),
    "catboost": cb.CatBoostRegressor(
        **{**modelos()["catboost"].get_params(), "iterations": k},
        cat_features=["sector"]),
}
filas = []
for nombre, m in cat.items():
    m.fit(X.loc[ent, COL], yr[ent])
    sin = ic(predice(nombre, ajustados[nombre], X.loc[val, FE], k),
             val).mean()
    con = ic(m.predict(X.loc[val, COL]), val).mean()
    filas.append({"modelo": nombre, "ic_val_sin_sector": sin,
                  "ic_val_con_sector": con})
pd.DataFrame(filas).set_index("modelo").round(4)

          ic_val_sin_sector  ic_val_con_sector
modelo
xgboost              0.0077             0.0043
lightgbm             0.0070             0.0039
catboost             0.0088             0.0077

With 200 trees, adding the sector lowers XGBoost's validation IC from 0.0077 to 0.0043 and LightGBM's from 0.0070 to 0.0039: almost by half. CatBoost also gets worse, but less (from 0.0088 to 0.0077). The figure adds the result of exercise 2: the same LightGBM with the sector in ten one-hot columns barely changes (0.0068). The next cell shows why.

cortes = pd.Series(cat["lightgbm"].booster_.feature_importance("split"),
                   index=COL)
print(f"splits on sector: {cortes['sector']} of {cortes.sum()}")
cortes.sort_values(ascending=False).head(6)

splits on sector: 104 of 547
sector      104
ret_1s       77
ruido_5      35
mom_12_1     34
sue          31
ret_1m       27
dtype: int32

The sector is the feature with the most splits in the model, 104 of 547, ahead of ret_1s (77), the strongest effect in the panel. A partition of ten categories chosen with the objective itself almost always finds something that improves the training fit: with 511 ways of splitting ten sectors into two groups, there is always one that matches the noise of those weeks. Behind it appears ruido_5, fat-tailed noise, with 35 splits. A categorical with many levels and no effect is a source of overfitting; if it has to be used, it is better with CatBoost's protection, as one-hot or as a statistic computed outside the training sample.

Determinism and speed

The course requires every output to be identical on every run. The three libraries allow it on CPU with the settings in the table; the cell checks it by fitting each model twice.

for nombre in ["xgboost", "lightgbm", "catboost"]:
    a = modelos()[nombre]
    a.set_params(**{"iterations" if nombre == "catboost"
                    else "n_estimators": 100})
    p1 = a.fit(X.loc[ent, FE], yr[ent]).predict(X.loc[val, FE])
    p2 = a.fit(X.loc[ent, FE], yr[ent]).predict(X.loc[val, FE])
    print(f"{nombre:9s} identical: {np.array_equal(p1, p2)}")

xgboost   identical: True
lightgbm  identical: True
catboost  identical: True

Identical does not mean the same on another machine: with another CPU or another number of threads, floating-point sums can change in the last decimals and, on a split that sits right at the limit, change a tree. That is why the course outputs are rounded to what the conclusion needs, and in production the trained model is stored instead of retraining it to reproduce a prediction. On GPU (device="cuda" in XGBoost, task_type="GPU" in CatBoost) there are operations that do not guarantee the same summation order; the course only uses CPU.

Speed is easy to measure and hard to compare, because it depends on the machine, the threads and the load. The code to measure it, not run in the course (its timings would change on every run):

import time

for nombre, m in modelos().items():
    t0 = time.perf_counter()
    m.fit(X.loc[tr_todo, FE], yr[tr_todo])
    print(f"{nombre:9s} {time.perf_counter() - t0:5.1f} s")

On the machine where the lesson was written (16 cores shared with other processes, 4 threads per model), 800 trees with 145,200 rows took about 3 s in XGBoost, 4 s in LightGBM and 4-4.5 s in CatBoost; Ridge, 0.04 s. With deterministic=True and force_row_wise=True LightGBM gives up part of its speed, and with depth-2 trees the advantage of leaf-wise growth does not show. Published speed comparisons use other data and other settings: the only sure thing is that all three are fast enough to retrain every week and that Ridge is a hundred times faster.

Common mistakes

• Using the default values. Depth 6 or 31 leaves and a rate of 0.1-0.3 are designed for problems with signal; on returns they overfit within a few trees.
• Stopping on the loss instead of the objective's metric. The squared error chose 14 trees; the IC, 25. And in any case, purged validation.
• Choosing the number of trees on the test. The 2019-2025 curve is not looked at to decide; that is what the validation is for. After choosing, retrain with training and validation together.
• Adding categoricals with no effect. The sector takes more splits than the best feature and halves the IC.
• Forgetting the baseline. An IC of 0.0160 looks good until you see that Ridge, over the same weeks and in a fraction of the time, gives 0.0180.
• Half-way reproducibility. Without deterministic=True and force_row_wise=True in LightGBM, or without allow_writing_files=False in CatBoost, the result or the working directory changes on every run.

Exercises

1. The comparison with Ridge was made with a single partition. Repeat it with an annual walk-forward from 2015 to 2025 (retraining every January on everything before it, with a one-week purge) for LightGBM with 25 trees. Does it hold?

anio = pd.DatetimeIndex(sem).year.to_numpy()
p_lg = np.full(len(d), np.nan)
p_r = np.full(len(d), np.nan)
for a in range(2015, 2026):
    tr = sem < fe(f"{a}-01-01") - np.timedelta64(7, "D")
    t_ = anio == a
    m = modelos()["lightgbm"].set_params(n_estimators=25)
    p_lg[t_] = m.fit(X.loc[tr, FE], yr[tr]).predict(X.loc[t_, FE])
    p_r[t_] = (Ridge(alpha=10.0).fit(X.loc[tr, FE], yr[tr])
               .predict(X.loc[t_, FE]))
wf = anio >= 2015
i_lg, i_r = ic(p_lg[wf], wf), ic(p_r[wf], wf)
dif = i_lg - i_r
print(f"LightGBM {i_lg.mean():.4f}  Ridge {i_r.mean():.4f}  "
      f"difference {dif.mean():+.4f} ± "
      f"{dif.std() / np.sqrt(len(dif)):.4f}")

LightGBM 0.0137  Ridge 0.0178  difference -0.0041 ± 0.0018

2. Encode the sector in ten one-hot columns (pd.get_dummies) and repeat the LightGBM measurement with 200 trees. Why does one-hot do less damage than the native categorical?

uno = pd.get_dummies(d["sector"], prefix="s", dtype="float32")
Xo = pd.concat([X[FE], uno], axis=1)
m = modelos()["lightgbm"].set_params(n_estimators=200)
m.fit(Xo[ent], yr[ent])
print(f"validation IC with sector as one-hot: "
      f"{ic(m.predict(Xo[val]), val).mean():.4f}")

validation IC with sector as one-hot: 0.0068

3. No code: a colleague replaces the production Ridge with a LightGBM because “in the backtest the IC rises from 0.018 to 0.031”. The LightGBM has 31 leaves, 1,000 trees, a rate of 0.1, the sector and the industry as categoricals, and the number of trees was chosen with early stopping on the last two years of the backtest. What would you check?