Decision trees, Random Forest and bagging.

What a tree does

A regression tree is a sequence of yes-or-no questions about the features. At each node it searches, among all the features and all possible thresholds, for the split $x_j \le c$ that most reduces the squared error of the two halves:

where $I$ and $D$ are the rows that end up on the left and on the right. It repeats the process in each half until a stopping rule is met (maximum depth, minimum rows per leaf) and predicts the mean of the label in each leaf. The result is a step function: constant in each rectangle of the feature space.

Three properties make it attractive for financial data:

• Invariant to monotonic transformations. A split at vol_63d ≤ 0.3 is the same as one on its logarithm or on its rank. No scaling is needed.
• It captures interactions and nonlinearities without being asked. A split on ret_1s followed by another on volumen_rel is an interaction; a split on sue is a threshold. A linear model needs someone to write those terms.
• It is fast. With the histogram algorithm of the boosting libraries (lesson 5.5.3), hundreds of thousands of rows are fitted in seconds.

And one flaw that decides everything: very high variance. Each split is chosen with the node's data, and in the deep nodes few rows remain. With a label in which the signal is 0.1% of the variance (lesson 5.5.1), a large tree spends its time fitting noise. Bagging and Random Forest exist to correct that.

The test bench

The data are those of the level's synthetic panel (400 fictitious stocks, 730 weeks, 21 weekly features, the ground truth published in public/curso/datos/nivel-5/README.md); nothing that comes out here is evidence about real markets. As in lesson 5.5.1, the features are in normal ranks within each week. Here the label is too: the normal rank of the following week's return. A tree that fits means is sensitive to the tails (a stock that rises 40% drags the mean of its leaf), and turning the label into ranks neutralizes them without changing what is evaluated, the weekly rank IC.

So that the lesson runs in under two minutes, a single split is used: training 2012-2018 without the last week (its label ends in 2019: the purge of lesson 5.4.2) and test 2019-2025, 365 weeks. In production you would run an annual walk-forward like the one in lesson 5.5.1, with more trees and a hyperparameter grid validated inside each training period.

import numpy as np
import pandas as pd
from scipy.stats import norm
from sklearn.linear_model import LinearRegression

f = pd.read_parquet("panel5_features.parquet")
e = pd.read_parquet("panel5_etiquetas.parquet")
v = pd.read_parquet("panel5_verdad.parquet")
d = (f.merge(e[["fecha", "ticker", "ret_sig_semana", "ret_sig_21d"]],
             on=["fecha", "ticker"])
      .merge(v, on=["fecha", "ticker"])
      .dropna(subset=["ret_sig_semana"])
      .reset_index(drop=True))
FE = [c for c in f.columns[3:] if c != "revision_analistas"]

def g(s):
    """Normal rank within each week (lesson 5.2.4); missing, 0."""
    por = s.groupby(d["fecha"])
    u = (por.rank() - 0.5) / por.transform("count")
    return pd.Series(norm.ppf(u), index=s.index).fillna(0.0)

def ic(score, mask):
    """Rank IC of each week between score and the label."""
    t = pd.DataFrame({"s": score, "f": sem[mask],
                      "y": d.loc[mask, "ret_sig_semana"].to_numpy()})
    t[["s", "y"]] = t.groupby("f")[["s", "y"]].rank()
    return t.groupby("f")[["s", "y"]].corr().xs("s", level=1)["y"]

X = np.column_stack([g(d[k]) for k in FE]).astype(np.float32)
yr = g(d["ret_sig_semana"]).to_numpy()       # label in ranks
sem = d["fecha"].to_numpy()
tr = sem < np.datetime64("2018-12-24")       # 2012-2018, purged
te = sem >= np.datetime64("2019-01-01")      # 2019-2025
mco = LinearRegression().fit(X[tr], yr[tr])
ic_te = ic(mco.predict(X[te]), te)
se = ic_te.std() / np.sqrt(len(ic_te))
print(f"OLS: IC in sample {ic(mco.predict(X[tr]), tr).mean():.4f}  "
      f"out of sample {ic_te.mean():.4f} ± {se:.4f}")

OLS: IC in sample 0.0274  out of sample 0.0180 ± 0.0045

The baseline: OLS with the 21 features gives an IC of 0.0180 ± 0.0045 out of sample (0.0274 in sample). It is the figure every model in this lesson has to beat.

A small tree, read from top to bottom

from sklearn.tree import DecisionTreeRegressor, export_text

arbol = DecisionTreeRegressor(max_depth=2, random_state=0)
arbol.fit(X[tr], yr[tr])
print(export_text(arbol, feature_names=FE, decimals=3))
filas_hoja = pd.Series(arbol.apply(X[tr])).value_counts().sort_index()
print("rows per leaf:", filas_hoja.tolist())

|--- ret_1s <= 0.512
|   |--- mom_12_1 <= -0.708
|   |   |--- value: [-0.021]
|   |--- mom_12_1 >  -0.708
|   |   |--- value: [0.020]
|--- ret_1s >  0.512
|   |--- ruido_1 <= -2.304
|   |   |--- value: [0.158]
|   |--- ruido_1 >  -2.304
|   |   |--- value: [-0.025]

rows per leaf: [24016, 76956, 431, 43797]

flowchart TB
  A{"ret_1s ≤ 0.512"} -->|yes| B{"mom_12_1 ≤ −0.708"}
  A -->|no| C{"ruido_1 ≤ −2.304"}
  B -->|yes| D["−0.021"]
  B -->|no| E["0.020"]
  C -->|yes| F["0.158"]
  C -->|no| G["−0.025"]

The first split is the strongest effect in the panel: the stocks that rose the most during the week (rank of ret_1s above 0.512, the top 30%) tend to turn around. In the left branch, the second split separates by momentum, the second effect. In the right one, the tree splits on ruido_1, pure noise by construction: 431 rows (0.3% of the training set) with extreme values of ruido_1 had, by chance, a high mean in 2012-2018 (0.158 in ranks), and that is enough for that split to be the one that most reduces the error in that node. A tree does not know which features make sense; it only knows which split reduces the error on the rows in front of it.

Depth and leaf size

filas = []
for prof in [1, 2, 3, 4, 6, 8, 12, None]:
    a = DecisionTreeRegressor(max_depth=prof, random_state=0)
    a.fit(X[tr], yr[tr])
    filas.append({"profundidad": prof or "no limit",
                  "hojas": a.get_n_leaves(),
                  "ic_dentro": ic(a.predict(X[tr]), tr).mean(),
                  "ic_fuera": ic(a.predict(X[te]), te).mean()})
pd.DataFrame(filas).round(4)

  profundidad   hojas  ic_dentro  ic_fuera
0           1       2     0.0156    0.0130
1           2       4     0.0222    0.0121
2           3       8     0.0246    0.0124
3           4      16     0.0300    0.0104
4           6      63     0.0416    0.0081
5           8     215     0.0581    0.0098
6          12    1417     0.1156    0.0084
7    no limit  142712     1.0000    0.0024

The figure at the top draws this table. The in-sample IC rises with each level, from 0.0156 with a single split to 1.0000 with no limit, when each of the 142,712 leaves has a single row and the prediction is the label itself. Out of sample the opposite happens: the maximum is at depth 1 (0.0130) and with no limit it stays at 0.0024. No tree reaches OLS (0.0180). With this signal-to-noise ratio, the optimal tree is a stump: one split on the feature with the most signal.

Depth is not the only way to rein in the tree. The minimum leaf size (min_samples_leaf) lets the tree grow where there are many rows and stops it where few remain.

filas = []
for hoja in [100, 1000, 3000, 10000]:
    a = DecisionTreeRegressor(min_samples_leaf=hoja, random_state=0)
    a.fit(X[tr], yr[tr])
    filas.append({"min_hoja": hoja, "hojas": a.get_n_leaves(),
                  "profundidad": a.get_depth(),
                  "ic_dentro": ic(a.predict(X[tr]), tr).mean(),
                  "ic_fuera": ic(a.predict(X[te]), te).mean()})
pd.DataFrame(filas).round(4)

   min_hoja  hojas  profundidad  ic_dentro  ic_fuera
0       100   1116           42     0.2337   -0.0013
1      1000    114           16     0.0761    0.0092
2      3000     37           13     0.0449    0.0093
3     10000     12            6     0.0305    0.0111

With leaves of at least 100 rows the tree reaches 42 levels and 1,116 leaves, and out of sample the IC is negative (−0.0013). With 10,000 rows per leaf it stays at 12 leaves and 0.0111. Leaf size is the hyperparameter that matters most on noisy data, because it sets how many observations each prediction averages: with noise of standard deviation 1 (the label in ranks) and a signal of the order of 0.03, thousands of rows are needed for the mean of one leaf to tell it apart from another. That is why in finance leaves of hundreds or thousands of rows are used, not the 1 to 5 of textbook examples.

Bagging and Random Forest

A large tree has low bias and high variance. Breiman's (1996) idea is to reduce the variance by averaging: train $B$ trees, each with a different bootstrap sample of the rows, and average their predictions (bagging, from bootstrap aggregating). If each tree has variance $\sigma^2$ and the correlation between two trees is $\rho$, the variance of the mean is

As trees are added the second term disappears, but the first does not: the floor is $\rho\sigma^2$. Random Forest (Breiman, 2001) lowers $\rho$ by forcing each split to choose only among max_features randomly drawn features. If the strongest feature is not in the draw, the tree splits on another one, and the trees look less alike.

flowchart TB
  D[Rows from 2012-2018] --> B1[Sample 1]
  D --> B2[Sample 2]
  D --> B3[Sample B]
  B1 --> T1[Tree 1]
  B2 --> T2[Tree 2]
  B3 --> T3[Tree B]
  T1 --> M[Mean]
  T2 --> M
  T3 --> M

In scikit-learn, max_features is the number (or fraction) of candidate features at each split: with 21 it is pure bagging; the classic rule for regression was one third (7); "sqrt" gives 4. max_samples=0.3 makes each tree see a bootstrap sample of 30% of the rows: faster and more different trees. The cell trains four forests of 100 trees with leaves of at least 1,000 rows, stores the prediction of each tree separately and measures the mean correlation between trees, $\rho$, and the IC when averaging the first $b$ trees.

from sklearn.ensemble import RandomForestRegressor

def bosque(mf, n=100, hoja=1000):
    return RandomForestRegressor(
        n_estimators=n, min_samples_leaf=hoja, max_features=mf,
        max_samples=0.3, n_jobs=4, random_state=0)

curvas, filas, bosques = {}, [], {}
for mf in [21, 7, 4, 1]:                       # 21 = pure bagging
    rf = bosques[mf] = bosque(mf).fit(X[tr], yr[tr])
    P = np.stack([t.predict(X[te]) for t in rf.estimators_])
    rho = np.corrcoef(P[:30])[np.triu_indices(30, 1)].mean()
    curvas[mf] = [ic(P[:b].mean(0), te).mean()
                  for b in [1, 3, 10, 30, 100]]
    filas.append({"max_features": mf, "rho_arboles": rho,
                  "ic_dentro": ic(rf.predict(X[tr]), tr).mean(),
                  "ic_1_arbol": curvas[mf][0],
                  "ic_100": curvas[mf][-1]})
pd.DataFrame(filas).round(4)

   max_features  rho_arboles  ic_dentro  ic_1_arbol  ic_100
0            21       0.0932     0.0735      0.0031  0.0130
1             7       0.0823     0.0672     -0.0014  0.0137
2             4       0.0770     0.0651      0.0039  0.0136
3             1       0.0519     0.0618      0.0034  0.0123

random_state=0 with n_jobs=4 is deterministic: each tree receives its own seed derived from the forest's and the split across threads does not change the result. What the table shows:

• A single tree is worthless: IC between −0.0014 and 0.0039 out of sample. The mean of 100 reaches 0.0123-0.0137. It is the variance reduction of the term $(1-\rho)\sigma^2/B$.
• The correlation between trees is low in every case (from 0.093 with bagging to 0.052 with one feature per split): with 30% samples and such a noisy label, each tree fits a different noise. Lowering $\rho$ with max_features helps little here, because the floor $\rho\sigma^2$ is already small. On data with more signal the effect is larger.
• The optimum is in between: 7 features per split (0.0137). With only one, each tree often splits on a feature with no effect and is worse (0.0123).

pd.DataFrame(curvas, index=[1, 3, 10, 30, 100]).round(4)

         21      7       4       1
1    0.0031 -0.0014  0.0039  0.0034
3    0.0047  0.0034  0.0063  0.0019
10   0.0105  0.0085  0.0090  0.0083
30   0.0114  0.0138  0.0130  0.0096
100  0.0130  0.0137  0.0136  0.0123

The IC rises quickly up to 30 trees and then flattens: adding trees never makes a forest worse (it does not overfit through the number of trees, unlike the boosting of lesson 5.5.3), but it stops helping once the $1/B$ term is already small. And no forest reaches the OLS line.

dif = ic(bosques[7].predict(X[te]), te) - ic_te
se_d = dif.std() / np.sqrt(len(dif))
print(f"forest (max_features=7) − OLS: {dif.mean():+.4f} ± {se_d:.4f}")

forest (max_features=7) − OLS: -0.0042 ± 0.0018

The paired difference, week by week, is −0.0042 ± 0.0018 (t ≈ −2.3): the forest is worse than the linear regression, and with a difference that is not noise. The panel's ground truth is almost linear in the ranks (61% of the alpha variance, lesson 5.5.1), and a forest approximates a straight line with steps that cost variance. It is what you would expect with this signal-to-noise ratio: a flexible model only wins if the nonlinear part of the signal makes up for the variance it adds.

The planted interaction: reversal × volume

The panel has a true interaction: the week's reversal is multiplied by $(1 - 0.5\,g(\text{volumen\_rel}))$. With low relative volume ($g = -1.2$) the reversal is 1.6 times the average; with high volume ($g = 1.2$), 0.4 times. It is a planted effect with known parameters, not a fact about markets: Campbell, Grossman and Wang (1993) find in indices and large US stocks, with daily data, the opposite relationship (the autocorrelation of returns falls with volume: more reversal when volume is high, which they attribute to liquidity pressure). What matters here is that the true form is known and you can check whether a model finds it.

The tool to see whether a model has captured it is partial dependence: ret_1s and volumen_rel are fixed on a grid of values, the other features are left as they are in a sample of test rows and the prediction is averaged. The slope of the curve in ret_1s for each volume level measures the reversal the model sees. The cell compares three forests: one trained on the true alpha (without noise) and only those two features, which shows the best a forest can see; one with the label and the two features; and one with the label and all 21.

ir, iv = FE.index("ret_1s"), FE.index("volumen_rel")
gr = np.linspace(-2, 2, 9)                   # ranks of ret_1s
gv = [-1.2, 0.0, 1.2]                        # low/medium/high volume

def dependencia(modelo, Xm, cols):
    """Partial dependence: mean prediction fixing the two ranks."""
    a, b = cols.index(ir), cols.index(iv)
    z = np.empty((len(gr), len(gv)))
    for j, v_ in enumerate(gv):
        for i, r_ in enumerate(gr):
            Z = Xm.copy()
            Z[:, a], Z[:, b] = r_, v_
            z[i, j] = modelo.predict(Z).mean()
    return z

dos = [ir, iv]
alfa = 1e4 * (d["alfa_verdadero"]
              - d.groupby("fecha")["alfa_verdadero"].transform("mean"))
casos = {"truth (alpha)": (alfa.to_numpy(), dos, None),
         "forest, 2 features": (yr, dos, None),
         "forest, 21 features": (yr, list(range(21)), 7)}
dep = {}
muestra = np.flatnonzero(te)[::50]           # 2,920 test rows
for nombre, (obj, cols, mf) in casos.items():
    rf = bosque(mf, hoja=2000).fit(X[tr][:, cols], obj[tr])
    dep[nombre] = dependencia(rf, X[muestra][:, cols], cols)
    pend = (dep[nombre][-1] - dep[nombre][0]) / 4
    print(f"{nombre:20s} relative slope (low, medium, high):",
          (pend / pend[1]).round(2))

truth (alpha)        relative slope (low, medium, high): [1.12 1.   0.35]
forest, 2 features   relative slope (low, medium, high): [1.12 1.   0.39]
forest, 21 features  relative slope (low, medium, high): [1.   1.   0.91]

Three results, one per panel of the figure:

• The forest trained on the truth sees a relative reversal of 1.12 / 1 / 0.35 with low, medium and high volume. The formula says 1.6 / 1 / 0.4: with 2,000-row leaves the forest smooths the extremes, but the shape is there.
• The forest with the label and only the two features gives 1.12 / 1 / 0.39, practically the same. With 145,000 rows and the chance to split on both, the forest finds the interaction despite the noise.
• The forest with the 21 features gives 1 / 1 / 0.91: it barely sees it. volumen_rel has no effect of its own, so a split on it does not reduce the error by itself and almost never wins the split contest against 20 rivals, several with a direct effect. Without splits on volume there is no interaction.

This is the uncomfortable lesson of trees in finance: they can represent any interaction, but they only find the ones that leave a trace in single-feature splits. The cheap alternative is to write it.

Xi = np.column_stack([X, X[:, ir] * X[:, iv]])
mi = LinearRegression().fit(Xi[tr], yr[tr])
b_r, b_i = mi.coef_[ir], mi.coef_[-1]
rel = [(b_r - 1.2 * b_i) / b_r, 1, (b_r + 1.2 * b_i) / b_r]
print("relative slope (low, medium, high):", np.round(rel, 2))
print(f"IC out of sample: {ic(mi.predict(Xi[te]), te).mean():.4f}")

relative slope (low, medium, high): [1.78 1.   0.22]
IC out of sample: 0.0192

OLS with a single added term, ret_1s × volumen_rel, sees 1.78 / 1 / 0.22 (the truth: 1.6 / 1 / 0.4) and gives an IC of 0.0192, the best of the lesson. One term with a hypothesis behind it is worth more than 100 trees searching blindly. Lesson 5.5.1 showed the other side: adding the 210 possible products without a hypothesis cost OLS a third of its IC.

The out-of-bag error is a random k-fold

Each tree in a forest leaves part of the rows out of its bootstrap sample; predicting each row only with the trees that did not see it gives the out-of-bag (OOB) error, a free validation. On independent data it is a good estimate of the generalization error. On financial series it is a shuffled k-fold: the rows a tree did not see are neighbors in time of rows it did see. With overlapping labels, that is leakage. The cell reproduces it with the setup from the data README: classify whether the 21-session return ends above the week's median, on 80 stocks, with a forest of small leaves.

from sklearn.ensemble import RandomForestClassifier

tick = np.sort(d["ticker"].unique())[::5]       # 80 stocks
m = d["ticker"].isin(tick).to_numpy() & d["ret_sig_21d"].notna()
m = m.to_numpy()
med = d.groupby("fecha")["ret_sig_21d"].transform("median")
z21 = (d["ret_sig_21d"] > med).to_numpy()[m].astype(int)
X80, s80 = X[m], sem[m]
corte = np.datetime64("2019-01-01")
a_tr = s80 < corte - np.timedelta64(28, "D")    # 4-week purge
rfc = RandomForestClassifier(
    n_estimators=60, min_samples_leaf=5, max_samples=0.3,
    oob_score=True, n_jobs=4, random_state=0).fit(X80[a_tr], z21[a_tr])
acierto = (rfc.predict(X80[s80 >= corte]) == z21[s80 >= corte]).mean()
print(f"OOB hit rate: {100 * rfc.oob_score_:.1f}%   "
      f"walk-forward 2019-2025: {100 * acierto:.1f}%")

OOB hit rate: 61.3%   walk-forward 2019-2025: 50.2%

The OOB promises a 61.3% hit rate; training up to 2018 with a four-week purge and predicting 2019-2025, it stays at 50.2%, the oracle level for this panel (the README gives 50.5% for the sign of the true alpha in that period). The eleven points of difference are memory: a week's 21-session label shares four weeks with the next week's, and the slow features barely change. It is the leakage of lesson 5.4.1 under another name. The fixes are those of that lesson and of 5.4.2 (purged validation), or the sequential bootstrap of lesson 5.3.4, which draws the rows with probability inverse to their overlap.

Impurity importance: what it does not say

Each split reduces the impurity (the squared error) of the node; summing those reductions per feature and normalizing gives the impurity importance (MDI) that scikit-learn stores in feature_importances_.

mdi = pd.Series(bosques[7].feature_importances_, index=FE)
mdi.sort_values(ascending=False).round(3).head(10)

ret_1s      0.074
mom_12_1    0.067
rsi_14      0.059
ret_3m      0.054
sue         0.053
vol_63d     0.050
ruido_2     0.049
ruido_4     0.049
vol_21d     0.048
ret_1m      0.047
dtype: float64

The first two are the strongest effects, but the rest is an almost flat list between 0.047 and 0.059 in which ruido_2 and ruido_4 tie with vol_63d and beat valor_bp, which has a true effect of 3.6 bp and does not appear in the top ten. MDI is computed in sample and rewards continuous features that offer many thresholds to fit noise; on a noisy problem that is mostly what it measures. For importance you have to use out-of-sample data and methods such as MDA, SFI or SHAP (lesson 5.2.5).

Common mistakes

• Small leaves. The scikit-learn defaults (min_samples_leaf=1, no depth limit) are designed for problems with signal. On returns, leaves of hundreds or thousands of rows.
• Validating with the OOB or with a shuffled k-fold. With overlapping labels it promises hit rates that do not exist (61.3% versus 50.2%). Purged validation, always.
• Expecting the forest to find every interaction. It finds the ones that leave a trace in single-feature splits. The ones with a hypothesis behind them are written.
• Reading impurity importance as evidence. It is an in-sample measure biased toward features with many values; noise comes out in the top ten.
• Not comparing with the linear baseline. A forest with an IC of 0.0137 looks like a good result until it is put next to an OLS of 0.0180 over the same weeks.
• Forgetting the cost. A forest of 100 trees with 1,000-row leaves takes tens of seconds where OLS takes milliseconds, and its prediction is not explained with a table of coefficients.

Exercises

1. Repeat the max_features table with 3,000-row leaves instead of 1,000 (only bagging and max_features=7, so that it runs quickly). Does the out-of-sample IC go up or down? And the correlation between trees?

for mf in [21, 7]:
    rf = bosque(mf, hoja=3000).fit(X[tr], yr[tr])
    P = np.stack([t.predict(X[te]) for t in rf.estimators_[:30]])
    rho = np.corrcoef(P)[np.triu_indices(30, 1)].mean()
    print(f"max_features {mf:2d}: rho {rho:.3f}  "
          f"IC out of sample {ic(rf.predict(X[te]), te).mean():.4f}")

max_features 21: rho 0.170  IC out of sample 0.0144
max_features  7: rho 0.159  IC out of sample 0.0151

2. Train a forest with only three features: ret_1s, volumen_rel and mom_12_1, and measure its out-of-sample IC and the relative slope of the reversal with low, medium and high volume. What do you gain and what do you lose compared with the 21-feature forest?

tres = [ir, iv, FE.index("mom_12_1")]
rf3 = bosque(None, hoja=2000).fit(X[tr][:, tres], yr[tr])
z3 = dependencia(rf3, X[muestra][:, tres], tres)
pend = (z3[-1] - z3[0]) / 4
print("relative slope:", (pend / pend[1]).round(2))
print(f"IC out of sample: {ic(rf3.predict(X[te][:, tres]), te).mean():.4f}")

relative slope: [0.96 1.   0.54]
IC out of sample: 0.0131

3. No code: a signal vendor presents a random forest on 150 daily features with “an out-of-bag hit rate of 58%” and an importance table in which the first is a social media sentiment feature. Which two numbers would you ask for, and why does the importance table not convince you?