--- title: "5.5.1 Ridge, Lasso, Elastic Net and logistic regression" description: "Ridge, Lasso, Elastic Net and logistic regression on a panel with published truth: regularization paths, selected noise and honest walk-forward." url: "https://pythonparatrading.com/en/python-trading-course/level-5/ridge-lasso-elastic-net-logistic-regression/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/ridge-lasso-elastic-net-logistic-regression.png" --- # 5.5.1 · Regularized regressions: Ridge, Lasso, Elastic Net and logistic > Penalized linear models are the baseline of all cross-sectional ML: fast, stable and hard to beat. This lesson fits them on the level's synthetic panel, whose true coefficients are known, and measures what they recover, what noise they select and how much they improve on OLS when the validation is honest. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.5 · Classical models - Duration: 70 min - Prerequisites: [2.3.5 · Factors (value, quality, low volatility, size) and event strategies](https://pythonparatrading.com/en/python-trading-course/level-2/factor-investing-and-event-strategies/), [2.5.2 · In-sample and out-of-sample: anchored and rolling walk-forward](https://pythonparatrading.com/en/python-trading-course/level-2/in-sample-out-of-sample-walk-forward/), [2.5.3 · Combinatorial purged cross-validation for strategies](https://pythonparatrading.com/en/python-trading-course/level-2/combinatorial-purged-cross-validation-cpcv/), [5.2.4 · Leakage-free normalization and cross-sectional rank](https://pythonparatrading.com/en/python-trading-course/level-5/leakage-free-normalization-cross-sectional-rank/), [5.4.2 · Purged k-fold with embargo and CPCV for ML models](https://pythonparatrading.com/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv/), [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/) ## By the end of this lesson - Write the objective functions of OLS, Ridge, Lasso, Elastic Net and penalized logistic regression, and translate scikit-learn's strength α into units that make sense. - Compare the estimated coefficients with the true ones and identify the three typical failures: collinearity, selected noise and confounding that no penalty corrects. - Read a regularization path and check whether the Lasso's selection repeats in other periods and with the same sign. - Choose α with a purged validation inside an annual walk-forward and compare the models by their IC with a standard error and paired differences. - Use logistic regression for direction and understand why its probabilities barely move away from 50%. ## Why penalize a linear model A cross-sectional linear model predicts the following week's return of each stock as a weighted sum of its features. It is the simplest model you can write and, on financial data, the hardest to beat: the signal is small, almost linear in the ranks of the features and changes over time, and a flexible model spends its flexibility fitting noise. That is why regularized regressions are the baseline of the whole module: every model in lessons 5.5.2 to 5.7.5 has to beat them out of sample to justify its cost. With $n$ observations, the feature matrix $X$ (standardized) and the label $y$, the five variants of this lesson minimize, as scikit-learn 1.9 writes them: $$ \begin{aligned} \text{OLS:}\quad & \lVert y - Xw \rVert_2^2 \\ \text{Ridge:}\quad & \lVert y - Xw \rVert_2^2 + \alpha \lVert w \rVert_2^2 \\ \text{Lasso:}\quad & \tfrac{1}{2n}\lVert y - Xw \rVert_2^2 + \alpha \lVert w \rVert_1 \\ \text{Elastic Net:}\quad & \tfrac{1}{2n}\lVert y - Xw \rVert_2^2 + \alpha\rho \lVert w \rVert_1 + \tfrac{\alpha(1-\rho)}{2} \lVert w \rVert_2^2 \end{aligned} $$ where $\rho$ is `l1_ratio` (0.5 by default in `ElasticNet`). Logistic regression models the probability that the stock ends up above the median, $p = 1/(1 + e^{-x'w})$, minimizes the log-loss and is penalized with `C` (the inverse of the strength) and `l1_ratio`. Two scikit-learn details worth knowing before touching α: - **Ridge does not divide the loss by $n$; Lasso and Elastic Net do.** With orthogonal features of variance 1, Ridge multiplies each OLS coefficient by $1/(1 + \alpha/n)$: with 145,600 rows, `alpha=1e5` leaves it at 59%. Lasso applies **soft thresholding**: it subtracts α from its absolute value and sets it to zero if it does not reach it, $w_j = \operatorname{sign}(\hat w_j)\max(|\hat w_j| - \alpha, 0)$. If the label is in basis points, so is the Lasso's α: `alpha=1` takes 1 bp off each coefficient. - **In logistic regression, `penalty` has been deprecated since 1.8** and disappears in 1.10: the penalty is chosen with `l1_ratio` (0 = L2, 1 = L1) and `C=np.inf` removes the penalty. Old code with `penalty="l1"` emits a `FutureWarning`. Geometry explains the difference between the two penalties. Ridge's is a round ball: the optimum touches the ball at any point, with all the coefficients non-zero and smaller. The Lasso's is a diamond with corners on the axes: the optimum tends to land on a corner, where several coefficients are exactly zero. That is why Ridge **shrinks** and Lasso **selects**. Elastic Net has corners (it selects) and curved sides (it spreads the weight among correlated features instead of keeping one at random). ```mermaid flowchart TB A[Features] --> B{Highly correlated} B -->|yes| C{Need to choose} B -->|no| D[OLS or mild Ridge] C -->|no| E[Ridge] C -->|yes| F[Elastic Net] A --> G{Many candidates} G -->|yes| H[Lasso or ENet] ``` None of the three replaces what really decides the result: features known at their date (lesson 5.2.2), leakage-free normalization (5.2.4) and a validation that respects time (5.4.2 and 5.4.3). ## The test bench: the panel with published truth The data are **synthetic**: 400 fictitious stocks, 730 weeks from 2012 to 2025 and 21 weekly features, generated with a model whose effects are published in `public/curso/datos/nivel-5/README.md`. The expected excess return of the following week (`alfa_verdadero`, in `panel5_verdad.parquet`) depends on six effects with coefficients of the order of 5 bp per week: momentum, one-week reversal (stronger with low volume), value, earnings surprise that decays, low volatility and sentiment. Another 14 features have no effect of their own, five of them pure noise. The leak `revision_analistas` is left out (exercise 1 uses it). Nothing that comes out here is evidence about real markets; it is a laboratory where you know what each method should find. The features are transformed into **cross-sectional normal ranks** each week (lesson 5.2.4), with 0 when the data point is missing (`sentimiento` is missing in 32% of the rows), and the label is the following week's return minus the mean of its week, in basis points. ```python import numpy as np import pandas as pd from scipy.stats import norm f = pd.read_parquet("panel5_features.parquet") e = pd.read_parquet("panel5_etiquetas.parquet") v = pd.read_parquet("panel5_verdad.parquet") d = (f.merge(e[["fecha", "ticker", "ret_sig_semana"]], on=["fecha", "ticker"]) .merge(v, on=["fecha", "ticker"]) .dropna(subset=["ret_sig_semana"]) .reset_index(drop=True)) FE = [c for c in f.columns[3:] if c != "revision_analistas"] def g(s): """Normal rank within each week (lesson 5.2.4); missing, 0.""" por = s.groupby(d["fecha"]) u = (por.rank() - 0.5) / por.transform("count") return pd.Series(norm.ppf(u), index=s.index).fillna(0.0) def sin_media(s): return s - s.groupby(d["fecha"]).transform("mean") G = pd.DataFrame({k: g(d[k]) for k in FE}) y = 1e4 * sin_media(d["ret_sig_semana"]) # bp per week sem = d["fecha"].to_numpy() ent = sem < np.datetime64("2019-01-01") # 2012-2018 print(G.shape, "training rows:", ent.sum()) print(f"standard deviation of y: {y[ent].std():.0f} bp") ``` ```output (291600, 21) training rows: 145600 standard deviation of y: 365 bp ``` The label has a standard deviation of 365 bp per week and the effects are about 5 bp per unit of feature: a signal-to-noise ratio of one to seventy. To know what a linear model should recover in this representation, the next cell projects the true alpha onto the 21 features. Those coefficients are the **linear truth**: the best a linear model can do on the ranks of these features. ```python from sklearn.linear_model import LinearRegression alfa = 1e4 * sin_media(d["alfa_verdadero"]) proy = LinearRegression().fit(G[ent], alfa[ent]) verdad = pd.Series(proy.coef_, index=FE) print(f"R² of the alpha on G: {proy.score(G[ent], alfa[ent]):.2f}") print(f"R² of the oracle on y: " f"{alfa[ent].var() / y[ent].var():.4f}") verdad[verdad.abs() > 0.3].round(2) ``` ```output R² of the alpha on G: 0.61 R² of the oracle on y: 0.0010 ret_1s -5.13 ret_1m 0.44 mom_12_1 4.65 vol_63d -3.73 dist_max_52s -0.49 valor_bp 3.59 ey 0.41 roe -0.36 sue 2.92 sentimiento 2.93 dtype: float64 ``` Two figures to keep in mind throughout the lesson. The first: a linear model on the ranks explains 61% of the variance of the true alpha; the rest is the interaction of reversal with volume, the decay of the SUE, the saturation of sentiment, the customer-graph component (lesson 5.7.4) and the weekly variation of the coefficients. The second: the **oracle**, which knows the exact alpha, explains 0.10% of the variance of the label. An out-of-sample R² of 1% on this panel would not be a great model: it would be a leak. ## OLS against the truth With 145,600 rows, OLS has a lot of information. The standard error has to be computed properly: the 400 stocks of the same week share the shocks of their sector, so the rows are not independent. The cell clusters the errors by week (*cluster* standard errors, lesson 2.1.1). ```python import statsmodels.api as sm mco = sm.OLS(y[ent].to_numpy(), sm.add_constant(G[ent])).fit( cov_type="cluster", cov_kwds={"groups": sem[ent]}) tabla = pd.DataFrame({"truth": verdad, "ols": mco.params[FE], "t": mco.tvalues[FE]}) tabla.round(2) ``` ```output truth ols t ret_1s -5.13 -5.10 -2.25 ret_1m 0.44 2.36 0.80 ret_3m 0.16 -1.99 -0.82 mom_12_1 4.65 5.29 2.23 vol_21d -0.01 -1.26 -0.52 vol_63d -3.73 1.76 0.69 volumen_rel -0.00 -0.20 -0.20 rsi_14 0.02 -2.12 -0.85 dist_max_52s -0.49 -0.65 -0.26 log_capitalizacion 0.04 -0.45 -0.43 valor_bp 3.59 5.90 2.77 ey 0.41 -4.34 -1.41 roe -0.36 2.42 1.09 sue 2.92 2.04 1.74 dias_desde_informe 0.02 0.23 0.23 sentimiento 2.93 1.91 1.64 ruido_1 -0.00 -1.25 -1.26 ruido_2 -0.00 1.27 1.40 ruido_3 0.07 -0.15 -0.17 ruido_4 -0.00 -0.02 -0.02 ruido_5 -0.05 -0.24 -0.26 ``` *Fig. — OLS coefficients of the 21 features against the linear projection of the true alpha, with their 95% interval clustered by week (synthetic panel, 2012-2018).* One row per feature. The hollow square is the truth and the filled one the OLS estimate, with its interval. ret_1s (truth −5.1, OLS −5.1) and mom_12_1 (4.7 and 5.3) are recovered well; valor_bp comes out at 5.9 with truth 3.6 and ey at −4.3 with truth 0.4, offsetting each other through collinearity. vol_63d (highlighted) has truth −3.7 and OLS +1.8: the beta of volatile stocks hides the premium. The intervals are between 3.5 and 12 basis points wide. The figure sums up the table. Three readings: - **The large effects are recovered.** `ret_1s` comes out at −5.10 with a true −5.13 and `mom_12_1` at 5.29 with 4.65. But with clustered errors the t of reversal is −2.25, not −5 as the naive formula would give: seven years of data give less independent information than 145,600 rows suggest. - **Collinearity scrambles the coefficients.** `ey` comes out at −4.34 (truth 0.41), `valor_bp` at 5.90 (truth 3.59) and `roe` at 2.42 (truth −0.36). The three offset each other: raising one and lowering another barely changes the fit. - **Confounding is not fixed with statistics.** `vol_63d` has a true effect of −3.73 and OLS estimates it at +1.76. Volatile stocks have more beta and the synthetic market rises on average, so the label (which only removes the week's mean, not beta) rewards volatility. It is the same effect the data README documents with Fama-MacBeth. No penalty corrects it; it takes another label (the return net of beta) or another feature. ```python print(G.loc[ent, ["valor_bp", "ey", "roe", "sue"]].corr().round(2)) print(G.loc[ent, ["ret_1s", "rsi_14", "mom_12_1", "ruido_4"]] .corr().round(2)) ``` ```output valor_bp ey roe sue valor_bp 1.00 0.71 0.16 0.02 ey 0.71 1.00 0.72 0.18 roe 0.16 0.72 1.00 0.29 sue 0.02 0.18 0.29 1.00 ret_1s rsi_14 mom_12_1 ruido_4 ret_1s 1.00 0.55 0.01 0.00 rsi_14 0.55 1.00 0.02 0.01 mom_12_1 0.01 0.02 1.00 0.60 ruido_4 0.00 0.01 0.60 1.00 ``` `ey` (earnings over market cap) is almost the product of `valor_bp` (book equity over market cap) and `roe` (earnings over book equity): correlations of 0.71 and 0.72. `rsi_14` is a version of reversal (0.55 with `ret_1s`) and `ruido_4` was built as 0.6 × momentum plus noise (0.60). These are the groups where an unpenalized linear model spreads the weight badly. ## Ridge and Lasso: shrink or select The next cell fits Ridge with $\alpha = 0.5\,n$ (each coefficient multiplied by about two thirds if the features were orthogonal) and Lasso with $\alpha = 1$ (a threshold of 1 bp). ```python from sklearn.linear_model import Lasso, Ridge n = ent.sum() X, Y = G[ent].to_numpy(), y[ent].to_numpy() tabla = tabla[["truth", "ols"]].copy() tabla["ridge"] = Ridge(alpha=0.5 * n).fit(X, Y).coef_ tabla["lasso"] = Lasso(alpha=1.0).fit(X, Y).coef_ tabla.round(2) ``` ```output truth ols ridge lasso ret_1s -5.13 -5.10 -3.25 -4.41 ret_1m 0.44 2.36 -0.05 -0.00 ret_3m 0.16 -1.99 -0.76 -0.09 mom_12_1 4.65 5.29 2.40 2.92 vol_21d -0.01 -1.26 -0.13 0.00 vol_63d -3.73 1.76 0.81 0.12 volumen_rel -0.00 -0.20 -0.17 -0.00 rsi_14 0.02 -2.12 -1.42 -0.78 dist_max_52s -0.49 -0.65 0.09 -0.00 log_capitalizacion 0.04 -0.45 -0.28 -0.00 valor_bp 3.59 5.90 1.81 2.12 ey 0.41 -4.34 0.19 0.00 roe -0.36 2.42 0.24 0.00 sue 2.92 2.04 1.54 1.27 dias_desde_informe 0.02 0.23 0.19 0.00 sentimiento 2.93 1.91 1.10 0.42 ruido_1 -0.00 -1.25 -0.83 -0.24 ruido_2 -0.00 1.27 0.86 0.28 ruido_3 0.07 -0.15 -0.13 -0.00 ruido_4 -0.00 -0.02 0.63 0.00 ruido_5 -0.05 -0.24 -0.15 -0.00 ``` Ridge undoes the offsetting within the value group: `ey` goes from −4.34 to 0.19 and `roe` from 2.42 to 0.24, close to their true values. In exchange, everything shrinks (reversal drops to −3.25) and the **grouping effect** appears: `ruido_4`, which OLS left at −0.02, receives 0.63 because Ridge spreads the momentum weight between the two correlated features (`mom_12_1` falls to 2.40). For a ranking it does not matter that everything shrinks; what matters are the proportions. Lasso sets 11 of the 21 features to zero, among them `ey` and `roe`. But it does not get the list right: it keeps `ruido_1` (−0.24) and `ruido_2` (0.28), pure noise; `rsi_14` (−0.78), which only copies reversal; and `vol_63d` with the wrong sign (0.12). And it shrinks `sentimiento` a lot (0.42 against a truth of 2.93). ```python from sklearn.linear_model import lasso_path alfas = np.geomspace(6, 0.05, 40) _, camino, _ = lasso_path(X - X.mean(0), Y - Y.mean(), alphas=alfas) camino = pd.DataFrame(camino.T, index=alfas.round(3), columns=FE) entra = (camino.abs() > 0).idxmax() # first α, from largest to smallest entra[camino.abs().max() > 0].sort_values(ascending=False).head(12) ``` ```output ret_1s 5.307 mom_12_1 3.248 sue 2.541 valor_bp 2.541 rsi_14 1.988 ruido_2 1.216 ruido_1 1.216 sentimiento 1.216 vol_63d 1.076 ret_3m 0.952 ret_1m 0.356 log_capitalizacion 0.356 dtype: float64 ``` The **regularization path** runs α from largest to smallest and records when each feature enters. The figure at the top draws seven of those paths. The strongest effects enter first (`ret_1s` at α = 5.3; `mom_12_1` at 3.2; `sue` and `valor_bp` at 2.5), then `rsi_14` (2.0), which has no effect of its own but resembles reversal, and at α = 1.2 **`ruido_1`, `ruido_2` and `sentimiento`** enter together: the Lasso does not distinguish a true 3 bp effect from a chance correlation in seven years of data. `vol_63d` enters afterwards, with the sign of beta. ## The Lasso selects noise That a feature enters the model does not say it has an effect. A cheap way to check is to repeat the selection in different periods and see whether it repeats and **with the same sign**. The cell fits the same Lasso (α = 1.5) in seven two-year windows, from 2012-2013 to 2024-2025. ```python anio = pd.DatetimeIndex(sem).year.to_numpy() signos = {} for a0 in range(2012, 2026, 2): m = (anio >= a0) & (anio <= a0 + 1) las = Lasso(alpha=1.5).fit(G[m], y[m]) signos[a0] = pd.Series(np.sign(las.coef_), index=FE) sg = pd.DataFrame(signos) cuenta = pd.DataFrame({"selected": (sg != 0).sum(axis=1), "positive": (sg > 0).sum(axis=1), "negative": (sg < 0).sum(axis=1), "effect": verdad.round(1)}) cuenta.sort_values("selected", ascending=False).head(12) ``` ```output selected positive negative effect ret_1s 6 0 6 -5.1 valor_bp 6 5 1 3.6 log_capitalizacion 5 3 2 0.0 sue 5 5 0 2.9 ret_3m 4 1 3 0.2 mom_12_1 4 4 0 4.7 vol_21d 4 3 1 -0.0 ruido_1 4 1 3 -0.0 ret_1m 3 2 1 0.4 sentimiento 3 3 0 2.9 dias_desde_informe 3 1 2 0.0 vol_63d 3 3 0 -3.7 ``` Counting how many times it enters is not enough: `log_capitalizacion`, with no effect, enters in 5 of the 7 windows, more than `mom_12_1` (4) or `sentimiento` (3). What separates them is the **sign**: the true effects keep it (`ret_1s`, 6 negative out of 6; `sue`, 5 positive out of 5; `mom_12_1`, 4 of 4; `sentimiento`, 3 of 3) and the features with no effect change it (`log_capitalizacion`, 3 and 2; `ruido_1`, 1 and 3; `dias_desde_informe`, 1 and 2). The exception is the usual one: `vol_63d` is stable (3 of 3)... with the wrong sign, because the confounding with beta is also stable. Stability rules out noise; it does not rule out a bias. With 21 features the problem is small. With many candidates, which is the norm when features are generated in bulk, it is serious. The next cell adds the 210 pairwise products of features (231 columns) and looks at what enters first. The only true interaction is `ret_1s*volumen_rel`: reversal is stronger with low volume. ```python from itertools import combinations pares = list(combinations(FE, 2)) nombres = FE + [f"{a}*{b}" for a, b in pares] X231 = np.column_stack( [G.to_numpy()] + [(G[a] * G[b]).to_numpy() for a, b in pares] ).astype(np.float32) Xc = X231[ent] - X231[ent].mean(0) al, cam, _ = lasso_path(Xc, Y - Y.mean(), precompute=True, alphas=np.geomspace(6, 1.5, 12)) cam = pd.DataFrame(cam.T, index=al.round(2), columns=nombres) orden = (cam.abs() > 0).idxmax()[cam.abs().max() > 0] orden.sort_values(ascending=False).head(14) ``` ```output ret_1s 5.29 mom_12_1 3.62 valor_bp 2.48 sue 2.48 rsi_14 2.19 ret_1m*volumen_rel 2.19 mom_12_1*ey 2.19 roe*dias_desde_informe 2.19 vol_63d*dias_desde_informe 2.19 vol_63d*ruido_1 2.19 valor_bp*ruido_5 2.19 sue*ruido_1 1.93 ret_1s*volumen_rel 1.93 ret_3m*mom_12_1 1.93 dtype: float64 ``` At α = 2.19 **six false interactions** enter at once (among them `vol_63d*ruido_1` and `valor_bp*ruido_5`, noise by construction) and the true one, `ret_1s*volumen_rel`, enters one step later, together with two other false ones. With 210 candidates, the maximum of 210 chance correlations is as large as a small true effect: it is the multiple testing problem of lesson 2.1.8 in another disguise. `float32` in `X231` is not a detail: the matrix takes 270 MB instead of 540, and `precompute=True` makes the Lasso work with $X'X$ (231 × 231) instead of going through the rows at each iteration. Does it matter out of sample? The cell trains on 2012-2018 (without the last week, whose label ends in 2019: the minimal purge of lesson 5.4.2) and measures the weekly IC from 2019 to 2025. ```python def ic(score, mask): """Rank IC of each week between score and the label.""" t = pd.DataFrame({"s": score, "f": sem[mask], "y": d.loc[mask, "ret_sig_semana"].to_numpy()}) t[["s", "y"]] = t.groupby("f")[["s", "y"]].rank() return t.groupby("f")[["s", "y"]].corr().xs("s", level=1)["y"] def resumen(serie): se = serie.std() / np.sqrt(len(serie)) return f"IC {serie.mean():.4f} ± {se:.4f} t {serie.mean()/se:4.1f}" purga = sem < np.datetime64("2018-12-24") # without the last week pru = sem >= np.datetime64("2019-01-01") Ytr = y[purga].to_numpy() for nombre, cols, mod in [ ("ols 21", slice(0, 21), LinearRegression()), ("ols 231", slice(None), LinearRegression()), ("ridge 231", slice(None), Ridge(alpha=2.0 * n)), ("lasso 231", slice(None), Lasso(alpha=2.0, precompute=True))]: mod.fit(X231[purga][:, cols], Ytr) s = mod.predict(X231[pru][:, cols]) print(f"{nombre:10s}", resumen(ic(s, pru))) print(f"{'oracle':10s}", resumen(ic(alfa[pru].to_numpy(), pru))) ``` ```output ols 21 IC 0.0195 ± 0.0045 t 4.4 ols 231 IC 0.0133 ± 0.0035 t 3.9 ridge 231 IC 0.0134 ± 0.0036 t 3.7 lasso 231 IC 0.0188 ± 0.0051 t 3.7 oracle IC 0.0256 ± 0.0052 t 4.9 ``` *Fig. — IC from 2019 to 2025 with 21 features and with 231 (the 21 and their 210 products), training on 2012-2018: OLS, Ridge and Lasso against the oracle.* Five rows with the mean weekly IC and its 95% interval. OLS with 21 features, 0.0195. OLS with 231 (highlighted) drops to 0.0133 and Ridge with 231 stays at 0.0134. Lasso with 231 recovers 0.0188, almost the same as the small model. The oracle gives 0.0256. Adding 210 candidates to OLS costs it a third of the IC (from 0.0195 to 0.0133). Ridge does not fix it (0.0134): it shrinks the 210 noise interactions but does not remove them, and their sum keeps moving the ranking. Lasso does recover almost all of it (0.0188) because it sets most of them to zero. None beats the small model. Regularization is insurance against a bad set of candidates, not a way of turning them into signal; the cheapest insurance is not to generate 210 candidates without a hypothesis behind them (lesson 2.3.5). ## Honest validation: walk-forward with α chosen without looking at the test Up to here α has been set by hand. In practice it is chosen with validation, and the validation has to respect time. The scheme of the next cell, for each year from 2015 to 2025: ```mermaid flowchart TB A[Data up to t−1] --> B[Sub-training] A --> C[Validate t−2 and t−1] B --> D[IC for each α] C --> D D --> E[Best α] E --> F[Retrain on everything] F --> G[Predict year t] ``` - **Training**: all the weeks before the year, except the last one (its label ends inside the test year). - **Validation**: the last two years of the training set; sub-training ends one week before they begin. - **Selection**: the α with the highest mean IC in validation; then the model is retrained on the whole training set and the year is predicted. It is a walk-forward with an expanding window (lesson 5.4.3) with a purged validation inside (lesson 5.4.2). The label covers one week, so purging one week is enough; with 21-session labels you would have to purge four. ```python from sklearn.linear_model import ElasticNet modelos = { "ols": (lambda a: LinearRegression(), [None]), "ridge": (lambda a: Ridge(alpha=a), [1e3, 1e4, 1e5, 1e6]), "lasso": (lambda a: Lasso(alpha=a), [0.1, 0.5, 1, 2, 3]), "enet": (lambda a: ElasticNet(alpha=a), [0.2, 1, 2, 4, 6]), } semana = np.timedelta64(7, "D") Xg, yg = G.to_numpy(), y.to_numpy() pred = {k: np.full(len(d), np.nan) for k in modelos} alfa_elegido = {k: [] for k in modelos} for a in range(2015, 2026): ini, ini_val = (np.datetime64(f"{a}-01-01"), np.datetime64(f"{a - 2}-01-01")) tr = sem < ini - semana # one-week purge sub = sem < ini_val - semana val = tr & (sem >= ini_val) # the last two years te = anio == a for k, (hacer, rejilla) in modelos.items(): mejor = rejilla[0] if len(rejilla) > 1: notas = [ic(hacer(r).fit(Xg[sub], yg[sub]) .predict(Xg[val]), val).mean() for r in rejilla] mejor = rejilla[int(np.argmax(notas))] alfa_elegido[k].append(mejor) pred[k][te] = hacer(mejor).fit(Xg[tr], yg[tr]).predict(Xg[te]) wf = anio >= 2015 ics = {k: ic(p[wf], wf) for k, p in pred.items()} ics["oracle"] = ic(alfa[wf].to_numpy(), wf) for k, s in ics.items(): print(f"{k:8s}", resumen(s)) ``` ```output ols IC 0.0184 ± 0.0038 t 4.9 ridge IC 0.0175 ± 0.0039 t 4.5 lasso IC 0.0185 ± 0.0039 t 4.7 enet IC 0.0175 ± 0.0040 t 4.4 oracle IC 0.0269 ± 0.0040 t 6.8 ``` *Fig. — Out-of-sample IC of the linear models in a yearly walk-forward 2015-2025, with α chosen each year on a purged two-year validation.* Six rows with the mean weekly IC and its 95% interval. OLS (highlighted) 0.0184; Ridge 0.0175; Lasso 0.0185; Elastic Net 0.0175; logistic 0.0160; oracle, the true alpha, 0.0269. The five models have intervals that overlap almost completely, from 0.010 to 0.026: regularisation does not improve on OLS. The oracle marks the ceiling. Over 574 test weeks, OLS gives an IC of 0.0184 ± 0.0038, the same as the reference ridge in the data README (0.0185). Ridge (0.0175), Lasso (0.0185) and Elastic Net (0.0175) are within one standard error. The oracle gives 0.0269: the linear model captures about two thirds of the signal that exists, and what is missing is not recovered by penalizing. The table of chosen α values explains why validation does not help. ```python pd.DataFrame(alfa_elegido, index=range(2015, 2026)).drop(columns="ols") ``` ```output ridge lasso enet 2015 100000.0 3.0 4.0 2016 1000000.0 3.0 4.0 2017 1000000.0 1.0 2.0 2018 10000.0 0.5 0.2 2019 1000.0 0.1 0.2 2020 1000.0 0.1 0.2 2021 10000.0 0.5 0.2 2022 1000000.0 1.0 2.0 2023 1000000.0 3.0 6.0 2024 1000.0 0.1 0.2 2025 1000.0 0.1 0.2 ``` Ridge's α jumps from the minimum to the maximum of the grid from one year to the next (1,000 in 2019-2020, 1,000,000 in 2022-2023). It is not instability in the method: the validation curve is almost flat. With $n \gg p$, the penalty barely changes the **direction** of the coefficient vector, and a rank IC depends only on the direction (multiplying all the predictions by 0.5 does not change any ranking). What validation picks is, above all, two years of noise. The fair comparison between models is paired: the same weeks, the week-by-week IC difference and its standard error. ```python for k in ["ridge", "lasso", "enet", "oracle"]: dif = ics[k] - ics["ols"] se = dif.std() / np.sqrt(len(dif)) print(f"{k:8s} {dif.mean():+.4f} ± {se:.4f} " f"t {dif.mean() / se:+.1f}") ``` ```output ridge -0.0010 ± 0.0010 t -1.0 lasso +0.0000 ± 0.0011 t +0.0 enet -0.0009 ± 0.0012 t -0.8 oracle +0.0085 ± 0.0042 t +2.0 ``` Pairing reduces the standard error from 0.0038 to 0.0010: the models get the same weeks wrong. Even so, no difference is significant; the oracle's barely is (t = 2.0), which gives an idea of how hard it is to **prove** out of sample even an improvement that exists. With five models tested, moreover, the best IC is biased upward; the correction for the number of tests (Deflated Sharpe) is lesson 5.4.4. ## Logistic regression: direction instead of size Logistic regression changes the question: instead of the return, the probability that the stock ends up **above the median** of its week. It has two advantages on financial data: the binary label ignores the tails (a +40% counts the same as a +1%) and the output is a probability that can be calibrated. The cost is throwing away the size information. ```python from sklearn.linear_model import LogisticRegression from sklearn.metrics import log_loss z = (d["ret_sig_semana"] > d.groupby("fecha")["ret_sig_semana"].transform("median")) z = z.astype(int).to_numpy() p = np.full(len(d), np.nan) for a in range(2015, 2026): tr = sem < np.datetime64(f"{a}-01-01") - semana te = anio == a lr = LogisticRegression(C=1.0, l1_ratio=0.0, max_iter=500) p[te] = lr.fit(Xg[tr], z[tr]).predict_proba(Xg[te])[:, 1] acierto = ((p[wf] > 0.5) == z[wf]).mean() print(f"hit rate {100 * acierto:.2f}% " f"log-loss {log_loss(z[wf], p[wf]):.5f} (chance {np.log(2):.5f})") print("percentiles 1, 50 and 99 of p:", np.percentile(p[wf], [1, 50, 99]).round(3)) print(resumen(ic(p[wf], wf))) ``` ```output hit rate 50.60% log-loss 0.69314 (chance 0.69315) percentiles 1, 50 and 99 of p: [0.471 0.5 0.527] IC 0.0160 ± 0.0036 t 4.4 ``` The hit rate is 50.60%; that of the oracle over the same period, according to the README, is 50.89%. The log-loss, 0.69314, is one hundred-thousandth below that of a coin (ln 2 = 0.69315), and 98% of the probabilities lie between 0.471 and 0.527. That is what a good model looks like when the true R² is 0.1%: almost indistinguishable from chance stock by stock, useful when 400 stocks are ranked at once every week for years. The IC, 0.0160, is somewhat below that of OLS (0.0184): the binary label throws away information. `C=1.0` with 150,000 rows is an almost negligible penalty: in scikit-learn's logistic regression the loss is a sum, not a mean, so the weight of the penalty falls with $n$. With `l1_ratio=1.0` (L1) you need `solver="saga"` or `"liblinear"`; both are slower at this size. ```python lr = LogisticRegression(C=1.0, l1_ratio=0.0, max_iter=500) lr.fit(Xg[ent], z[ent]) comp = pd.DataFrame({"ols_bp": tabla["ols"], "logit_x1000": 1000 * lr.coef_[0]}, index=FE) print(f"correlation: {comp.corr().iloc[0, 1]:.3f}") comp.loc[["ret_1s", "mom_12_1", "valor_bp", "sue", "ey"]].round(2) ``` ```output correlation: 0.933 ols_bp logit_x1000 ret_1s -5.10 -22.33 mom_12_1 5.29 22.88 valor_bp 5.90 30.85 sue 2.04 5.71 ey -4.34 -14.87 ``` The logistic and OLS coefficients have a correlation of 0.933: the two models see the same thing. The differences are where the tail matters: `sue` weighs relatively less in the logistic regression (5.71 against 22.33 for reversal, when in OLS it is 2.04 against 5.10), because a good part of its effect arrives in large jumps that the binary label treats like any other rise. ## Common mistakes - **Penalizing without standardizing.** Ridge and Lasso penalize all coefficients equally; if one feature is in millions and another in fractions of one, the penalty punishes the second. Use normal ranks or `StandardScaler` inside the pipeline (lesson 5.1.2), fitted only on the training set. - **Reading Ridge's α as the Lasso's.** In scikit-learn, one scales with $n$ and the other does not. An `alpha=1` in Ridge with 145,600 rows does almost nothing; in Lasso it takes 1 bp off each coefficient. - **Interpreting the Lasso's selection as evidence.** Whatever correlates with the residual in that sample enters, noise included. You have to repeat it in other periods and require the same sign. - **Choosing α with shuffled k-fold.** In this panel the labels are weekly and do not overlap, but with 21-session labels random k-fold promises 60% hit rates that do not exist (lesson 5.4.1). Always use purged validation. - **Expecting the penalty to correct confounding.** `vol_63d` comes out with the sign of beta in OLS, in Ridge, in Lasso and in all seven windows. It is a problem with the label, not with the estimator. - **Comparing models without pairing or counting the tests.** IC differences of 0.001 are noise; models are compared week by week and with the number of configurations tested in plain view (lesson 5.4.4). ## Exercises 1. Add `revision_analistas` (the leaky feature) to the 21 and fit the Lasso with α = 1 on 2012-2018. What coefficient does it get and at what α does it enter the path? Does regularization detect that the feature is a leak? **Solution.** ```python G["revision_analistas"] = g(d["revision_analistas"]) X22 = G[ent].to_numpy() las = Lasso(alpha=1.0).fit(X22, Y) print(f"coefficient of the leak: {las.coef_[-1]:.1f} bp " f"(ret_1s: {las.coef_[0]:.1f} bp)") al22, cam22, _ = lasso_path(X22 - X22.mean(0), Y - Y.mean(), alphas=np.geomspace(80, 5, 12)) print("α at which it enters:", al22[np.argmax(cam22[-1] != 0)].round(1), "| features at α = 5:", int((cam22[:, -1] != 0).sum())) G = G.drop(columns="revision_analistas") ``` ```output coefficient of the leak: 69.7 bp (ret_1s: -4.0 bp) α at which it enters: 62.2 | features at α = 5: 2 ``` The leak gets 69.7 bp, about 17 times the coefficient of reversal (−4.0 bp, which shrinks because the leak takes over part of its role), and enters the path at α = 62.2, when `ret_1s` entered at 5.3: at α = 5 there are only two features in the model. Regularization does exactly what it should: reward the feature that correlates most with the label. That is why it cannot detect a leak; on the contrary, it picks it first. The leak is only uncovered by auditing when each piece of data is known (lesson 5.2.2). 2. The panel's momentum reverses in market rebounds (momentum crash, data README). Fit OLS separately on the weeks of each regime (column `regimen` of `panel5_coef_verdad.csv`) over the whole period and compare the coefficient of `mom_12_1` with the mean of the true coefficient `c_mom` in those weeks. **Solution.** ```python cv = pd.read_csv("panel5_coef_verdad.csv", parse_dates=["fecha"]) reg = d["fecha"].map(cv.set_index("fecha")["regimen"]).to_numpy() cmom = d["fecha"].map(cv.set_index("fecha")["c_mom"]).to_numpy() for r in ["calma", "estres", "rebote"]: m = reg == r b = LinearRegression().fit(G[m], y[m]).coef_[FE.index("mom_12_1")] print(f"{r:7s} weeks {len(np.unique(sem[m])):3d} " f"OLS {b:6.2f} bp truth {1e4 * cmom[m].mean():6.2f} bp") ``` ```output calma weeks 575 OLS 6.47 bp truth 4.71 bp estres weeks 89 OLS 5.63 bp truth 2.02 bp rebote weeks 65 OLS -1.88 bp truth -10.44 bp ``` OLS recovers the sign in all three regimes, not the size: 6.47 bp in calm (truth 4.71), 5.63 in stress (2.02) and −1.88 in rebounds, where the truth is −10.44. With 65 rebound weeks the standard error is several basis points and the estimate ends up far from the truth. A global linear model averages the three regimes and, in a rebound, bets exactly the wrong way. There are two ways out: give the model the regime as a feature (with interactions, which is what the trees of lesson 5.5.2 do) or retrain more often with windows that react (lesson 5.4.3). Both cost variance: there are few weeks of each rare regime. 3. No code: a colleague presents a Lasso on 300 automatically generated features that selects 12 and gives a mean IC of 0.08 in a five-fold k-fold. What three things would you ask for before believing it? **Solution.** First, the validation: an IC of 0.08 is three times that of the oracle in this panel and, on real data, a figure like that is almost always a leak or a shuffled k-fold with overlapping labels. I would ask for a walk-forward with purging and embargo (lessons 5.4.2 and 5.4.3) with α chosen inside each training set, not with all the data. Second, stability: the 12 selected features in other periods, with their sign; with 300 candidates the Lasso picks noise easily, as it did here with 231. Third, the time audit of the 12 features (when each piece of data is known) and the total number of configurations tested, to correct the result with the Deflated Sharpe (lesson 5.4.4). Only then would it make sense to talk about costs and capacity. ## Self-assessment 1. In scikit-learn, `Ridge(alpha=1e5)` and `Lasso(alpha=1)` on 145,600 rows with features of variance 1. What does each one do, approximately, to an OLS coefficient of 5 bp if the features were orthogonal? - A. Both multiply it by 0.5 - B. Ridge sets it to zero and Lasso leaves it unchanged - C. Ridge multiplies it by 1/(1 + α/n) ≈ 0.59 and Lasso subtracts 1 bp from it (soft thresholding): it ends up at 4 bp - D. Neither changes it, because α is small compared with n *Answer C: Ridge multiplies it by 1/(1 + α/n) ≈ 0.59 and Lasso subtracts 1 bp from it (soft thresholding): it ends up at 4 bp.* The Ridge loss in scikit-learn is not divided by n, so its effective strength is α/n; the Lasso's is (by 2n), and with standardized features its α is in the units of the coefficient: it is the threshold subtracted from each one. 2. OLS estimates a coefficient of −4.3 bp for `ey` and 5.9 bp for `valor_bp`, with true values of 0.4 and 3.6. What best explains the error? - A. That `ey` has a leak - B. That more trees are needed - C. That value has no effect - D. Collinearity: the two features have a correlation of 0.71 and `ey` also resembles `roe` (0.72); OLS splits the signal with opposite signs that offset each other *Answer D: Collinearity: the two features have a correlation of 0.71 and `ey` also resembles `roe` (0.72); OLS splits the signal with opposite signs that offset each other.* With highly correlated features, OLS has high variance in the direction of their difference: one coefficient rises and the other falls without the fit changing much. Ridge corrects it (ey 0.19, valor_bp 1.81) and Lasso drops `ey` and `roe`. 3. On the Lasso path, `ruido_2` (pure noise) enters at the same α as `sentimiento` (a true effect). Which conclusion is correct? - A. `ruido_2` has an effect that the generator does not document - B. With this signal-to-noise ratio, the order of entry does not distinguish a small effect from a chance correlation in the sample; you have to check whether the selection and the sign repeat in other periods - C. α has to be lowered until it drops out - D. The Lasso is implemented wrong *Answer B: With this signal-to-noise ratio, the order of entry does not distinguish a small effect from a chance correlation in the sample; you have to check whether the selection and the sign repeat in other periods.* The Lasso selects whatever correlates most with the residual in that sample. In two-year windows, the true features keep their sign (ret_1s, 6 of 6 negative; sue, 5 of 5 positive) and the noise features change it (ruido_1: one positive and three negative). 4. In the 2015-2025 walk-forward, the α chosen by validation jumps from 1,000 to 1,000,000 in Ridge from one year to the next. What does it indicate? - A. That the validation curve is almost flat and noisy: the IC barely changes with α and the choice is almost random, which is why Ridge does not improve on OLS (difference −0.0010 ± 0.0010) - B. That the grid is wrong - C. That you have to retrain every week - D. That Ridge overfits a lot *Answer A: That the validation curve is almost flat and noisy: the IC barely changes with α and the choice is almost random, which is why Ridge does not improve on OLS (difference −0.0010 ± 0.0010).* With n ≫ p, the penalty barely changes the direction of the coefficient vector, which is the only thing that matters for a rank IC. Choosing α by optimizing a noisy IC is choosing at random. 5. The direction logistic regression has a hit rate of 50.60% and probabilities between 0.471 and 0.527 (percentiles 1 and 99). Is it a useless model? - A. Yes: such a low hit rate cannot be traded - B. Not necessarily: the oracle's hit rate is 50.89%; with 400 stocks per week, a small, stable hit rate gives an IC of 0.016 with t = 4.4. What you cannot do is read its probabilities as certainties - C. No: a hit rate of 50.6% is excellent in any case - D. Yes, because its log-loss is higher than that of a coin *Answer B: Not necessarily: the oracle's hit rate is 50.89%; with 400 stocks per week, a small, stable hit rate gives an IC of 0.016 with t = 4.4. What you cannot do is read its probabilities as certainties.* In cross-sectional ML what you trade is the ordering of many stocks at once, not the probability of one. The log-loss (0.69314) is just below that of a coin (0.69315): it is what you would expect when the true R² is 0.1%. ## Key points - Ridge shrinks all the coefficients and spreads the weight among correlated features; Lasso sets them to zero (it selects); Elastic Net mixes the two. In scikit-learn, Ridge uses α without dividing by n and Lasso with 1/2n. - With 145,600 rows and 21 features, OLS already recovers the large effects well (ret_1s −5.1 against a true −5.1). What fails is collinearity (ey, roe, valor_bp) and confounding (vol_63d comes out with the wrong sign because of beta), which no penalty corrects. - The Lasso selects noise: ruido_2 enters at the same time as sentimiento, and with 231 candidates six false interactions enter before the true one. A selection is only believed if it repeats in other periods with the same sign. - In a walk-forward with α chosen in a purged validation, no regularization improves on OLS with 21 features (IC 0.0184 against 0.0175-0.0185). Regularization helps when there are many candidates: with 231, Lasso recovers 0.0188 against 0.0133 for OLS. - The logistic regression gives the same ordering as the linear regression (coefficient correlation 0.93) with probabilities between 0.47 and 0.53. The ceiling is the oracle: IC 0.0269. ## Resources ### Research - [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min ### Documentation - [scikit-learn 1.9: linear models (Ridge, Lasso, Elastic Net, logistic regression)](https://scikit-learn.org/stable/modules/linear_model.html) · scikit-learn.org ### Books - [Hoerl, A. E. and Kennard, R. W. (1970): Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics 12(1), 55-67](https://doi.org/10.1080/00401706.1970.10488634) · doi.org - [Tibshirani, R. (1996): Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society B 58(1), 267-288](https://doi.org/10.1111/j.2517-6161.1996.tb02080.x) · doi.org - [Zou, H. and Hastie, T. (2005): Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society B 67(2), 301-320](https://doi.org/10.1111/j.1467-9868.2005.00503.x) · doi.org - [Gu, S., Kelly, B. and Xiu, D. (2020): Empirical Asset Pricing via Machine Learning. The Review of Financial Studies 33(5), 2223-2273](https://doi.org/10.1093/rfs/hhaa009) · doi.org ### Data - [Level 5 data: panel features (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet Previous lesson: [5.4.4 · Backtesting ML signals with costs and Deflated Sharpe](https://pythonparatrading.com/en/python-trading-course/level-5/backtesting-ml-signals-deflated-sharpe/) Next lesson: [5.5.2 · Decision trees, Random Forest and bagging](https://pythonparatrading.com/en/python-trading-course/level-5/decision-trees-random-forest-bagging/)