--- title: "5.5.2 Decision trees, Random Forest and bagging" description: "Decision trees, bagging and Random Forest on a panel with published ground truth: depth, leaves, max_features, the planted interaction and OOB." url: "https://pythonparatrading.com/en/python-trading-course/level-5/decision-trees-random-forest-bagging/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/decision-trees-random-forest-bagging.png" --- # 5.5.2 · Decision trees, Random Forest and bagging > A tree cuts the feature space into rectangles and predicts the mean of each one; a forest averages hundreds of different trees to lower their variance. This lesson fits them on the level's synthetic panel, measures how much they overfit with each hyperparameter, checks whether they find the planted interaction between reversal and volume and compares them with the linear baseline of lesson 5.5.1. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.5 · Classical models - Duration: 70 min - Prerequisites: [5.5.1 · Regularized regressions: Ridge, Lasso, Elastic Net and logistic](https://pythonparatrading.com/en/python-trading-course/level-5/ridge-lasso-elastic-net-logistic-regression/), [5.4.1 · Why random k-fold fails on financial data](https://pythonparatrading.com/en/python-trading-course/level-5/why-random-k-fold-fails-finance/), [5.4.2 · Purged k-fold with embargo and CPCV for ML models](https://pythonparatrading.com/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv/) ## By the end of this lesson - Explain how a regression tree grows (splits that reduce the squared error) and read a small tree fitted on financial data. - Control a tree's overfitting with the maximum depth and the minimum leaf size, measuring the IC in sample and out of sample. - Explain why averaging trees reduces variance (bagging) and how max_features lowers the correlation between them (Random Forest). - Check with partial dependence whether a model captures a known interaction, and why a forest with many features dilutes it. - Recognize that the out-of-bag error is a random k-fold and that impurity importance rewards noise. ## What a tree does A regression tree is a sequence of yes-or-no questions about the features. At each node it searches, among all the features and all possible thresholds, for the split $x_j \le c$ that most reduces the squared error of the two halves: $$ \min_{j,\,c}\; \sum_{i \in I} (y_i - \overline{y}_I)^2 + \sum_{i \in D} (y_i - \overline{y}_D)^2 $$ where $I$ and $D$ are the rows that end up on the left and on the right. It repeats the process in each half until a stopping rule is met (maximum depth, minimum rows per leaf) and predicts the **mean of the label in each leaf**. The result is a step function: constant in each rectangle of the feature space. Three properties make it attractive for financial data: - **Invariant to monotonic transformations.** A split at `vol_63d ≤ 0.3` is the same as one on its logarithm or on its rank. No scaling is needed. - **It captures interactions and nonlinearities without being asked.** A split on `ret_1s` followed by another on `volumen_rel` is an interaction; a split on `sue` is a threshold. A linear model needs someone to write those terms. - **It is fast.** With the histogram algorithm of the boosting libraries (lesson 5.5.3), hundreds of thousands of rows are fitted in seconds. And one flaw that decides everything: **very high variance**. Each split is chosen with the node's data, and in the deep nodes few rows remain. With a label in which the signal is 0.1% of the variance (lesson 5.5.1), a large tree spends its time fitting noise. Bagging and Random Forest exist to correct that. ## The test bench The data are those of the level's **synthetic** panel (400 fictitious stocks, 730 weeks, 21 weekly features, the ground truth published in `public/curso/datos/nivel-5/README.md`); nothing that comes out here is evidence about real markets. As in lesson 5.5.1, the features are in normal ranks within each week. Here the label is too: the normal rank of the following week's return. A tree that fits means is sensitive to the tails (a stock that rises 40% drags the mean of its leaf), and turning the label into ranks neutralizes them without changing what is evaluated, the weekly rank IC. So that the lesson runs in under two minutes, a single split is used: training 2012-2018 without the last week (its label ends in 2019: the purge of lesson 5.4.2) and test 2019-2025, 365 weeks. In production you would run an annual walk-forward like the one in lesson 5.5.1, with more trees and a hyperparameter grid validated inside each training period. ```python import numpy as np import pandas as pd from scipy.stats import norm from sklearn.linear_model import LinearRegression f = pd.read_parquet("panel5_features.parquet") e = pd.read_parquet("panel5_etiquetas.parquet") v = pd.read_parquet("panel5_verdad.parquet") d = (f.merge(e[["fecha", "ticker", "ret_sig_semana", "ret_sig_21d"]], on=["fecha", "ticker"]) .merge(v, on=["fecha", "ticker"]) .dropna(subset=["ret_sig_semana"]) .reset_index(drop=True)) FE = [c for c in f.columns[3:] if c != "revision_analistas"] def g(s): """Normal rank within each week (lesson 5.2.4); missing, 0.""" por = s.groupby(d["fecha"]) u = (por.rank() - 0.5) / por.transform("count") return pd.Series(norm.ppf(u), index=s.index).fillna(0.0) def ic(score, mask): """Rank IC of each week between score and the label.""" t = pd.DataFrame({"s": score, "f": sem[mask], "y": d.loc[mask, "ret_sig_semana"].to_numpy()}) t[["s", "y"]] = t.groupby("f")[["s", "y"]].rank() return t.groupby("f")[["s", "y"]].corr().xs("s", level=1)["y"] X = np.column_stack([g(d[k]) for k in FE]).astype(np.float32) yr = g(d["ret_sig_semana"]).to_numpy() # label in ranks sem = d["fecha"].to_numpy() tr = sem < np.datetime64("2018-12-24") # 2012-2018, purged te = sem >= np.datetime64("2019-01-01") # 2019-2025 mco = LinearRegression().fit(X[tr], yr[tr]) ic_te = ic(mco.predict(X[te]), te) se = ic_te.std() / np.sqrt(len(ic_te)) print(f"OLS: IC in sample {ic(mco.predict(X[tr]), tr).mean():.4f} " f"out of sample {ic_te.mean():.4f} ± {se:.4f}") ``` ```output OLS: IC in sample 0.0274 out of sample 0.0180 ± 0.0045 ``` The baseline: OLS with the 21 features gives an IC of 0.0180 ± 0.0045 out of sample (0.0274 in sample). It is the figure every model in this lesson has to beat. ## A small tree, read from top to bottom ```python from sklearn.tree import DecisionTreeRegressor, export_text arbol = DecisionTreeRegressor(max_depth=2, random_state=0) arbol.fit(X[tr], yr[tr]) print(export_text(arbol, feature_names=FE, decimals=3)) filas_hoja = pd.Series(arbol.apply(X[tr])).value_counts().sort_index() print("rows per leaf:", filas_hoja.tolist()) ``` ```output |--- ret_1s <= 0.512 | |--- mom_12_1 <= -0.708 | | |--- value: [-0.021] | |--- mom_12_1 > -0.708 | | |--- value: [0.020] |--- ret_1s > 0.512 | |--- ruido_1 <= -2.304 | | |--- value: [0.158] | |--- ruido_1 > -2.304 | | |--- value: [-0.025] rows per leaf: [24016, 76956, 431, 43797] ``` ```mermaid flowchart TB A{"ret_1s ≤ 0.512"} -->|yes| B{"mom_12_1 ≤ −0.708"} A -->|no| C{"ruido_1 ≤ −2.304"} B -->|yes| D["−0.021"] B -->|no| E["0.020"] C -->|yes| F["0.158"] C -->|no| G["−0.025"] ``` The first split is the strongest effect in the panel: the stocks that rose the most during the week (rank of `ret_1s` above 0.512, the top 30%) tend to turn around. In the left branch, the second split separates by momentum, the second effect. In the right one, the tree splits on **`ruido_1`**, pure noise by construction: 431 rows (0.3% of the training set) with extreme values of `ruido_1` had, by chance, a high mean in 2012-2018 (0.158 in ranks), and that is enough for that split to be the one that most reduces the error in that node. A tree does not know which features make sense; it only knows which split reduces the error on the rows in front of it. ## Depth and leaf size ```python filas = [] for prof in [1, 2, 3, 4, 6, 8, 12, None]: a = DecisionTreeRegressor(max_depth=prof, random_state=0) a.fit(X[tr], yr[tr]) filas.append({"profundidad": prof or "no limit", "hojas": a.get_n_leaves(), "ic_dentro": ic(a.predict(X[tr]), tr).mean(), "ic_fuera": ic(a.predict(X[te]), te).mean()}) pd.DataFrame(filas).round(4) ``` ```output profundidad hojas ic_dentro ic_fuera 0 1 2 0.0156 0.0130 1 2 4 0.0222 0.0121 2 3 8 0.0246 0.0124 3 4 16 0.0300 0.0104 4 6 63 0.0416 0.0081 5 8 215 0.0581 0.0098 6 12 1417 0.1156 0.0084 7 no limit 142712 1.0000 0.0024 ``` The figure at the top draws this table. The in-sample IC rises with each level, from 0.0156 with a single split to 1.0000 with no limit, when each of the 142,712 leaves has a single row and the prediction is the label itself. Out of sample the opposite happens: the maximum is at **depth 1** (0.0130) and with no limit it stays at 0.0024. No tree reaches OLS (0.0180). With this signal-to-noise ratio, the optimal tree is a stump: one split on the feature with the most signal. Depth is not the only way to rein in the tree. The **minimum leaf size** (`min_samples_leaf`) lets the tree grow where there are many rows and stops it where few remain. ```python filas = [] for hoja in [100, 1000, 3000, 10000]: a = DecisionTreeRegressor(min_samples_leaf=hoja, random_state=0) a.fit(X[tr], yr[tr]) filas.append({"min_hoja": hoja, "hojas": a.get_n_leaves(), "profundidad": a.get_depth(), "ic_dentro": ic(a.predict(X[tr]), tr).mean(), "ic_fuera": ic(a.predict(X[te]), te).mean()}) pd.DataFrame(filas).round(4) ``` ```output min_hoja hojas profundidad ic_dentro ic_fuera 0 100 1116 42 0.2337 -0.0013 1 1000 114 16 0.0761 0.0092 2 3000 37 13 0.0449 0.0093 3 10000 12 6 0.0305 0.0111 ``` With leaves of at least 100 rows the tree reaches 42 levels and 1,116 leaves, and out of sample the IC is negative (−0.0013). With 10,000 rows per leaf it stays at 12 leaves and 0.0111. Leaf size is the hyperparameter that matters most on noisy data, because it sets how many observations each prediction averages: with noise of standard deviation 1 (the label in ranks) and a signal of the order of 0.03, thousands of rows are needed for the mean of one leaf to tell it apart from another. That is why in finance leaves of hundreds or thousands of rows are used, not the 1 to 5 of textbook examples. ## Bagging and Random Forest A large tree has low bias and high variance. Breiman's (1996) idea is to reduce the variance by averaging: train $B$ trees, each with a different bootstrap sample of the rows, and average their predictions (**bagging**, from *bootstrap aggregating*). If each tree has variance $\sigma^2$ and the correlation between two trees is $\rho$, the variance of the mean is $$ \operatorname{Var}\Big(\frac{1}{B}\sum_{b=1}^{B} T_b(x)\Big) = \rho\,\sigma^2 + \frac{1-\rho}{B}\,\sigma^2 . $$ As trees are added the second term disappears, but the first does not: the floor is $\rho\sigma^2$. Random Forest (Breiman, 2001) lowers $\rho$ by forcing each split to choose only among `max_features` randomly drawn features. If the strongest feature is not in the draw, the tree splits on another one, and the trees look less alike. ```mermaid flowchart TB D[Rows from 2012-2018] --> B1[Sample 1] D --> B2[Sample 2] D --> B3[Sample B] B1 --> T1[Tree 1] B2 --> T2[Tree 2] B3 --> T3[Tree B] T1 --> M[Mean] T2 --> M T3 --> M ``` In scikit-learn, `max_features` is the number (or fraction) of candidate features at each split: with 21 it is pure bagging; the classic rule for regression was one third (7); `"sqrt"` gives 4. `max_samples=0.3` makes each tree see a bootstrap sample of 30% of the rows: faster and more different trees. The cell trains four forests of 100 trees with leaves of at least 1,000 rows, stores the prediction of each tree separately and measures the mean correlation between trees, $\rho$, and the IC when averaging the first $b$ trees. ```python from sklearn.ensemble import RandomForestRegressor def bosque(mf, n=100, hoja=1000): return RandomForestRegressor( n_estimators=n, min_samples_leaf=hoja, max_features=mf, max_samples=0.3, n_jobs=4, random_state=0) curvas, filas, bosques = {}, [], {} for mf in [21, 7, 4, 1]: # 21 = pure bagging rf = bosques[mf] = bosque(mf).fit(X[tr], yr[tr]) P = np.stack([t.predict(X[te]) for t in rf.estimators_]) rho = np.corrcoef(P[:30])[np.triu_indices(30, 1)].mean() curvas[mf] = [ic(P[:b].mean(0), te).mean() for b in [1, 3, 10, 30, 100]] filas.append({"max_features": mf, "rho_arboles": rho, "ic_dentro": ic(rf.predict(X[tr]), tr).mean(), "ic_1_arbol": curvas[mf][0], "ic_100": curvas[mf][-1]}) pd.DataFrame(filas).round(4) ``` ```output max_features rho_arboles ic_dentro ic_1_arbol ic_100 0 21 0.0932 0.0735 0.0031 0.0130 1 7 0.0823 0.0672 -0.0014 0.0137 2 4 0.0770 0.0651 0.0039 0.0136 3 1 0.0519 0.0618 0.0034 0.0123 ``` `random_state=0` with `n_jobs=4` is deterministic: each tree receives its own seed derived from the forest's and the split across threads does not change the result. What the table shows: - **A single tree is worthless**: IC between −0.0014 and 0.0039 out of sample. The mean of 100 reaches 0.0123-0.0137. It is the variance reduction of the term $(1-\rho)\sigma^2/B$. - **The correlation between trees is low in every case** (from 0.093 with bagging to 0.052 with one feature per split): with 30% samples and such a noisy label, each tree fits a different noise. Lowering $\rho$ with `max_features` helps little here, because the floor $\rho\sigma^2$ is already small. On data with more signal the effect is larger. - **The optimum is in between**: 7 features per split (0.0137). With only one, each tree often splits on a feature with no effect and is worse (0.0123). ```python pd.DataFrame(curvas, index=[1, 3, 10, 30, 100]).round(4) ``` ```output 21 7 4 1 1 0.0031 -0.0014 0.0039 0.0034 3 0.0047 0.0034 0.0063 0.0019 10 0.0105 0.0085 0.0090 0.0083 30 0.0114 0.0138 0.0130 0.0096 100 0.0130 0.0137 0.0136 0.0123 ``` *Fig. — Out-of-sample IC when averaging 1, 3, 10, 30 and 100 trees: bagging (all 21 features at each split) and Random Forest with 7 and 1 features per split, against OLS.* Three lines that rise with the number of trees. A single tree gives an IC between −0.001 and 0.004. With 100 trees, bagging reaches 0.0130, Random Forest with 7 features per split (highlighted) 0.0137 and with 1 feature 0.0123. None reaches the dashed OLS line, 0.0180. The IC rises quickly up to 30 trees and then flattens: adding trees never makes a forest worse (it does not overfit through the number of trees, unlike the boosting of lesson 5.5.3), but it stops helping once the $1/B$ term is already small. And no forest reaches the OLS line. ```python dif = ic(bosques[7].predict(X[te]), te) - ic_te se_d = dif.std() / np.sqrt(len(dif)) print(f"forest (max_features=7) − OLS: {dif.mean():+.4f} ± {se_d:.4f}") ``` ```output forest (max_features=7) − OLS: -0.0042 ± 0.0018 ``` The paired difference, week by week, is −0.0042 ± 0.0018 (t ≈ −2.3): the forest is **worse** than the linear regression, and with a difference that is not noise. The panel's ground truth is almost linear in the ranks (61% of the alpha variance, lesson 5.5.1), and a forest approximates a straight line with steps that cost variance. It is what you would expect with this signal-to-noise ratio: a flexible model only wins if the nonlinear part of the signal makes up for the variance it adds. ## The planted interaction: reversal × volume The panel has a true interaction: the week's reversal is multiplied by $(1 - 0.5\,g(\text{volumen\_rel}))$. With low relative volume ($g = -1.2$) the reversal is 1.6 times the average; with high volume ($g = 1.2$), 0.4 times. It is a planted effect with known parameters, not a fact about markets: Campbell, Grossman and Wang (1993) find in indices and large US stocks, with daily data, the opposite relationship (the autocorrelation of returns falls with volume: more reversal when volume is high, which they attribute to liquidity pressure). What matters here is that the true form is known and you can check whether a model finds it. The tool to see whether a model has captured it is **partial dependence**: `ret_1s` and `volumen_rel` are fixed on a grid of values, the other features are left as they are in a sample of test rows and the prediction is averaged. The slope of the curve in `ret_1s` for each volume level measures the reversal the model sees. The cell compares three forests: one trained on the **true alpha** (without noise) and only those two features, which shows the best a forest can see; one with the label and the two features; and one with the label and all 21. ```python ir, iv = FE.index("ret_1s"), FE.index("volumen_rel") gr = np.linspace(-2, 2, 9) # ranks of ret_1s gv = [-1.2, 0.0, 1.2] # low/medium/high volume def dependencia(modelo, Xm, cols): """Partial dependence: mean prediction fixing the two ranks.""" a, b = cols.index(ir), cols.index(iv) z = np.empty((len(gr), len(gv))) for j, v_ in enumerate(gv): for i, r_ in enumerate(gr): Z = Xm.copy() Z[:, a], Z[:, b] = r_, v_ z[i, j] = modelo.predict(Z).mean() return z dos = [ir, iv] alfa = 1e4 * (d["alfa_verdadero"] - d.groupby("fecha")["alfa_verdadero"].transform("mean")) casos = {"truth (alpha)": (alfa.to_numpy(), dos, None), "forest, 2 features": (yr, dos, None), "forest, 21 features": (yr, list(range(21)), 7)} dep = {} muestra = np.flatnonzero(te)[::50] # 2,920 test rows for nombre, (obj, cols, mf) in casos.items(): rf = bosque(mf, hoja=2000).fit(X[tr][:, cols], obj[tr]) dep[nombre] = dependencia(rf, X[muestra][:, cols], cols) pend = (dep[nombre][-1] - dep[nombre][0]) / 4 print(f"{nombre:20s} relative slope (low, medium, high):", (pend / pend[1]).round(2)) ``` ```output truth (alpha) relative slope (low, medium, high): [1.12 1. 0.35] forest, 2 features relative slope (low, medium, high): [1.12 1. 0.39] forest, 21 features relative slope (low, medium, high): [1. 1. 0.91] ``` *Fig. — Partial dependence on the ret_1s rank with low, mid and high relative volume: forest trained on the true alpha, on the label with two features, and on the label with all 21.* Three panels, each with three lines that fall from left to right (reversal): low volume (highlighted), mid and high (dashed). With the true alpha, the high-volume line is much flatter: reversal weakens with volume. The forest trained on the label with only ret_1s and volumen_rel reproduces that shape, with more noise. The forest with all 21 features gives three almost parallel lines: it does not see the interaction. Three results, one per panel of the figure: - **The forest trained on the truth** sees a relative reversal of 1.12 / 1 / 0.35 with low, medium and high volume. The formula says 1.6 / 1 / 0.4: with 2,000-row leaves the forest smooths the extremes, but the shape is there. - **The forest with the label and only the two features** gives 1.12 / 1 / 0.39, practically the same. With 145,000 rows and the chance to split on both, the forest finds the interaction despite the noise. - **The forest with the 21 features** gives 1 / 1 / 0.91: it barely sees it. `volumen_rel` has no effect of its own, so a split on it does not reduce the error by itself and almost never wins the split contest against 20 rivals, several with a direct effect. Without splits on volume there is no interaction. This is the uncomfortable lesson of trees in finance: they can represent any interaction, but they only **find** the ones that leave a trace in single-feature splits. The cheap alternative is to write it. ```python Xi = np.column_stack([X, X[:, ir] * X[:, iv]]) mi = LinearRegression().fit(Xi[tr], yr[tr]) b_r, b_i = mi.coef_[ir], mi.coef_[-1] rel = [(b_r - 1.2 * b_i) / b_r, 1, (b_r + 1.2 * b_i) / b_r] print("relative slope (low, medium, high):", np.round(rel, 2)) print(f"IC out of sample: {ic(mi.predict(Xi[te]), te).mean():.4f}") ``` ```output relative slope (low, medium, high): [1.78 1. 0.22] IC out of sample: 0.0192 ``` OLS with a single added term, `ret_1s × volumen_rel`, sees 1.78 / 1 / 0.22 (the truth: 1.6 / 1 / 0.4) and gives an IC of 0.0192, the best of the lesson. One term with a hypothesis behind it is worth more than 100 trees searching blindly. Lesson 5.5.1 showed the other side: adding the 210 possible products without a hypothesis cost OLS a third of its IC. ## The out-of-bag error is a random k-fold Each tree in a forest leaves part of the rows out of its bootstrap sample; predicting each row only with the trees that did not see it gives the **out-of-bag** (OOB) error, a free validation. On independent data it is a good estimate of the generalization error. On financial series it is a shuffled k-fold: the rows a tree did not see are neighbors in time of rows it did see. With overlapping labels, that is leakage. The cell reproduces it with the setup from the data README: classify whether the 21-session return ends above the week's median, on 80 stocks, with a forest of small leaves. ```python from sklearn.ensemble import RandomForestClassifier tick = np.sort(d["ticker"].unique())[::5] # 80 stocks m = d["ticker"].isin(tick).to_numpy() & d["ret_sig_21d"].notna() m = m.to_numpy() med = d.groupby("fecha")["ret_sig_21d"].transform("median") z21 = (d["ret_sig_21d"] > med).to_numpy()[m].astype(int) X80, s80 = X[m], sem[m] corte = np.datetime64("2019-01-01") a_tr = s80 < corte - np.timedelta64(28, "D") # 4-week purge rfc = RandomForestClassifier( n_estimators=60, min_samples_leaf=5, max_samples=0.3, oob_score=True, n_jobs=4, random_state=0).fit(X80[a_tr], z21[a_tr]) acierto = (rfc.predict(X80[s80 >= corte]) == z21[s80 >= corte]).mean() print(f"OOB hit rate: {100 * rfc.oob_score_:.1f}% " f"walk-forward 2019-2025: {100 * acierto:.1f}%") ``` ```output OOB hit rate: 61.3% walk-forward 2019-2025: 50.2% ``` The OOB promises a 61.3% hit rate; training up to 2018 with a four-week purge and predicting 2019-2025, it stays at 50.2%, the oracle level for this panel (the README gives 50.5% for the sign of the true alpha in that period). The eleven points of difference are memory: a week's 21-session label shares four weeks with the next week's, and the slow features barely change. It is the leakage of lesson 5.4.1 under another name. The fixes are those of that lesson and of 5.4.2 (purged validation), or the **sequential bootstrap** of lesson 5.3.4, which draws the rows with probability inverse to their overlap. ## Impurity importance: what it does not say Each split reduces the impurity (the squared error) of the node; summing those reductions per feature and normalizing gives the **impurity importance** (MDI) that scikit-learn stores in `feature_importances_`. ```python mdi = pd.Series(bosques[7].feature_importances_, index=FE) mdi.sort_values(ascending=False).round(3).head(10) ``` ```output ret_1s 0.074 mom_12_1 0.067 rsi_14 0.059 ret_3m 0.054 sue 0.053 vol_63d 0.050 ruido_2 0.049 ruido_4 0.049 vol_21d 0.048 ret_1m 0.047 dtype: float64 ``` The first two are the strongest effects, but the rest is an almost flat list between 0.047 and 0.059 in which `ruido_2` and `ruido_4` tie with `vol_63d` and beat `valor_bp`, which has a true effect of 3.6 bp and does not appear in the top ten. MDI is computed **in sample** and rewards continuous features that offer many thresholds to fit noise; on a noisy problem that is mostly what it measures. For importance you have to use out-of-sample data and methods such as MDA, SFI or SHAP (lesson 5.2.5). ## Common mistakes - **Small leaves.** The scikit-learn defaults (`min_samples_leaf=1`, no depth limit) are designed for problems with signal. On returns, leaves of hundreds or thousands of rows. - **Validating with the OOB or with a shuffled k-fold.** With overlapping labels it promises hit rates that do not exist (61.3% versus 50.2%). Purged validation, always. - **Expecting the forest to find every interaction.** It finds the ones that leave a trace in single-feature splits. The ones with a hypothesis behind them are written. - **Reading impurity importance as evidence.** It is an in-sample measure biased toward features with many values; noise comes out in the top ten. - **Not comparing with the linear baseline.** A forest with an IC of 0.0137 looks like a good result until it is put next to an OLS of 0.0180 over the same weeks. - **Forgetting the cost.** A forest of 100 trees with 1,000-row leaves takes tens of seconds where OLS takes milliseconds, and its prediction is not explained with a table of coefficients. ## Exercises 1. Repeat the `max_features` table with 3,000-row leaves instead of 1,000 (only bagging and `max_features=7`, so that it runs quickly). Does the out-of-sample IC go up or down? And the correlation between trees? **Solution.** ```python for mf in [21, 7]: rf = bosque(mf, hoja=3000).fit(X[tr], yr[tr]) P = np.stack([t.predict(X[te]) for t in rf.estimators_[:30]]) rho = np.corrcoef(P)[np.triu_indices(30, 1)].mean() print(f"max_features {mf:2d}: rho {rho:.3f} " f"IC out of sample {ic(rf.predict(X[te]), te).mean():.4f}") ``` ```output max_features 21: rho 0.170 IC out of sample 0.0144 max_features 7: rho 0.159 IC out of sample 0.0151 ``` With larger leaves each tree averages more rows and fits less noise, so it looks more like the others: the correlation between trees rises from 0.093 to 0.170 (bagging) and from 0.082 to 0.159 (7 features). The out-of-sample IC improves somewhat, from 0.0130 to 0.0144 and from 0.0137 to 0.0151, and stays below OLS (0.0180). Leaf size and the number of trees do the same job by two routes (averaging within each tree or across trees), and neither of them takes the forest up to OLS, because what it lacks is not only variance but a better functional form. Trying several leaf sizes and keeping the best one on 2019-2025 would, moreover, be choosing with the test data: the grid is validated inside the training period (lesson 5.5.4). 2. Train a forest with only three features: `ret_1s`, `volumen_rel` and `mom_12_1`, and measure its out-of-sample IC and the relative slope of the reversal with low, medium and high volume. What do you gain and what do you lose compared with the 21-feature forest? **Solution.** ```python tres = [ir, iv, FE.index("mom_12_1")] rf3 = bosque(None, hoja=2000).fit(X[tr][:, tres], yr[tr]) z3 = dependencia(rf3, X[muestra][:, tres], tres) pend = (z3[-1] - z3[0]) / 4 print("relative slope:", (pend / pend[1]).round(2)) print(f"IC out of sample: {ic(rf3.predict(X[te][:, tres]), te).mean():.4f}") ``` ```output relative slope: [0.96 1. 0.54] IC out of sample: 0.0131 ``` With three features the forest sees part of the interaction: with high volume the reversal drops to 0.54 of the average, although with low volume it does not rise (0.96), because `volumen_rel` now only competes with two rivals and the trees split on it now and then. What it loses is the signal from the rest of the true effects (value, SUE, sentiment, low volatility): its IC, 0.0131, falls below the 21-feature forest (0.0137) and OLS (0.0180). Selecting features with a hypothesis before training a forest is a way of regularizing it; doing it by looking at the out-of-sample result would be overfitting the selection. 3. No code: a signal vendor presents a random forest on 150 daily features with “an out-of-bag hit rate of 58%” and an importance table in which the first is a social media sentiment feature. Which two numbers would you ask for, and why does the importance table not convince you? **Solution.** The OOB is a random k-fold: with slow daily features and labels that usually span several days, the rows a tree did not see are almost copies of the ones it saw. I would ask for the hit rate (or the IC) in a walk-forward with purging and an embargo the size of the label horizon, and the paired comparison with a linear baseline over the same dates (lessons 5.4.2, 5.4.3 and 5.5.1). On this level's panel the OOB gives 61.3% and the walk-forward 50.2%. The importance table, if it is the one from `feature_importances_`, is an MDI computed in sample that rewards continuous features with many values, noise included; a feature coming out first does not mean it predicts out of sample. I would ask for the permutation importance (MDA) over the validation period or SHAP (lesson 5.2.5), and the result of the model without that feature. ## Self-assessment 1. A tree with no depth limit gives an IC of 1.00 in sample and 0.0024 out of sample. What happened? - A. The tree has found the true signal - B. There is leakage in the test data - C. Each of the 142,712 leaves contains a single row: the tree memorizes the training label, noise included, and out of sample it predicts almost at random - D. The in-sample IC is computed wrongly *Answer C: Each of the 142,712 leaves contains a single row: the tree memorizes the training label, noise included, and out of sample it predicts almost at random.* With one-row leaves, the in-sample prediction is the label itself. With a signal-to-noise ratio of one to seventy, almost everything memorized is noise; the best tree out of sample is the smallest one (depth 1, IC 0.0130). 2. Why does a Random Forest use a `max_features` smaller than the number of features? - A. To make the trees different from each other: the variance of a mean of B trees is ρσ² + (1 − ρ)σ²/B, and lowering the correlation ρ lowers the floor that adding trees does not remove - B. So that each tree trains faster, with no effect on the prediction - C. To remove the noise features - D. Because scikit-learn does not allow using all of them *Answer A: To make the trees different from each other: the variance of a mean of B trees is ρσ² + (1 − ρ)σ²/B, and lowering the correlation ρ lowers the floor that adding trees does not remove.* With all the features at each split (bagging), the trees almost always choose the same splits and look alike (ρ = 0.093 on the panel). With 1 feature per split, ρ drops to 0.052. The price is that each tree is worse; the optimum is in between (7 features: IC 0.0137). 3. A forest with only `ret_1s` and `volumen_rel` reproduces the planted interaction (relative slope 1.12 / 1 / 0.39), and one with the 21 features does not (1 / 1 / 0.91). Why? - A. Because the 21 features have a leak - B. Because `volumen_rel` has no effect of its own: with 21 candidates and a noisy label, the trees almost never split on it, and without splits on volume there is no interaction to show - C. Because forests cannot represent interactions - D. Because partial dependence is computed wrongly with 21 features *Answer B: Because `volumen_rel` has no effect of its own: with 21 candidates and a noisy label, the trees almost never split on it, and without splits on volume there is no interaction to show.* A tree only uses a feature if a split on it reduces the error in that node. A feature whose effect exists only in interaction with another does not reduce the error on its own and stays buried among 20 rivals; a linear model with the term `ret_1s × volumen_rel` written by hand recovers it better (1.78 / 1 / 0.22). 4. The classification random forest with 21-session labels gives an out-of-bag hit rate of 61.3% and 50.2% in the walk-forward. What explains the difference? - A. The market regime changed in 2019 - B. The OOB uses fewer trees - C. The walk-forward purge is too long - D. The rows outside a tree's bootstrap sample are neighbors in time of rows that are inside it, with overlapping labels: the OOB is a random k-fold and inherits its leakage *Answer D: The rows outside a tree's bootstrap sample are neighbors in time of rows that are inside it, with overlapping labels: the OOB is a random k-fold and inherits its leakage.* Each 21-session label shares four of its five weeks with the label of the following week. The tree that saw week t gets t + 1 right from memory, not from signal. It is the same leakage as in lesson 5.4.1. 5. In the forest's impurity importance (MDI), `ruido_2` and `ruido_4` appear among the top ten and `valor_bp` does not. How is it interpreted? - A. Noise is more useful than value - B. That `valor_bp` should be removed - C. MDI measures how much a feature is used for splitting in sample; continuous noise features offer many possible splits and absorb noise. It says nothing about their value out of sample - D. That the forest has found an effect in the noise *Answer C: MDI measures how much a feature is used for splitting in sample; continuous noise features offer many possible splits and absorb noise. It says nothing about their value out of sample.* Impurity importances are biased toward features with many values and are computed on the training data. MDA, SFI and SHAP on validation data (lesson 5.2.5) are the tools for importance. ## Key points - A regression tree splits the rows with the cut that most reduces the squared error and predicts the mean of each leaf. Without limits it memorizes: IC 1.00 in sample and 0.0024 out of sample. - With this signal-to-noise ratio, the best tree is tiny (depth 1: IC 0.0130) and none reaches the linear baseline (0.0180). Limiting the leaf to 10,000 rows gives 0.0111. - Averaging trees reduces variance: a single tree gives an IC between −0.001 and 0.004 and the mean of 100, from 0.012 to 0.014. max_features lowers the correlation between trees (from 0.093 to 0.052) and the optimum is in between (7 features: 0.0137). - A forest finds the reversal × volume interaction if it can split on both features, but with 21 candidates it dilutes it. Written by hand in a linear model it is recovered better and gives the best IC (0.0192). - On the panel, the best forest falls below OLS (−0.0042 ± 0.0018). The OOB promises a 61.3% hit rate with overlapping labels that the walk-forward leaves at 50.2%, and impurity importance puts noise among the top ten. ## Resources ### Documentation - [scikit-learn 1.9: decision trees](https://scikit-learn.org/stable/modules/tree.html) · scikit-learn.org - [scikit-learn 1.9: ensemble methods (bagging, Random Forest)](https://scikit-learn.org/stable/modules/ensemble.html) · scikit-learn.org ### Books - [Breiman, L. (1996): Bagging Predictors. Machine Learning 24, 123-140](https://doi.org/10.1007/BF00058655) · doi.org - [Breiman, L. (2001): Random Forests. Machine Learning 45, 5-32](https://doi.org/10.1023/A:1010933404324) · doi.org - [Hastie, T., Tibshirani, R. and Friedman, J. (2009): The Elements of Statistical Learning, 2nd ed., chapters 9, 10 and 15 (authors' PDF)](https://hastie.su.domains/ElemStatLearn/) · hastie.su.domains - [Campbell, J. Y., Grossman, S. J. and Wang, J. (1993): Trading Volume and Serial Correlation in Stock Returns. The Quarterly Journal of Economics 108(4), 905-939](https://doi.org/10.2307/2118454) · doi.org ### Data - [Level 5 data: panel features (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet Previous lesson: [5.5.1 · Regularized regressions: Ridge, Lasso, Elastic Net and logistic](https://pythonparatrading.com/en/python-trading-course/level-5/ridge-lasso-elastic-net-logistic-regression/) Next lesson: [5.5.3 · Gradient boosting: XGBoost, LightGBM and CatBoost](https://pythonparatrading.com/en/python-trading-course/level-5/xgboost-lightgbm-catboost-gradient-boosting/)