---
title: "7.6.4 Capstone C3: ML portfolio, HRP and Spanish taxes"
description: "Stock portfolio with a LightGBM ranking versus a ridge, HRP, rebalancing with costs, FIFO lots, Spain's two-month rule and IRPF on savings."
url: "https://pythonparatrading.com/en/python-trading-course/level-7/capstone-ml-portfolio-hrp-spanish-taxes/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-7/capstone-ml-portfolio-hrp-spanish-taxes.png"
---

# 7.6.4 · Capstone C3: stock and ETF portfolio with ML, HRP and Spanish taxation

> A long-term portfolio that ranks a stock universe with LightGBM every month, spreads risk with HRP and rebalances paying costs and Spain's IRPF on savings income, lot by lot. This guide is the brief of project C3, its architecture, a skeleton that works end to end with the level 5 panel and the way to validate it, run it on paper and defend it. It is not tax or investment advice.

- Course: [Level 7: Production, MLOps, regulation, career and capstone](https://pythonparatrading.com/en/python-trading-course/level-7/)
- Module: 7.6 · Final capstone
- Duration: 70 min
- Prerequisites: [7.6.1 · Capstone: common requirements, the panel's rubric and preparing the defense](https://pythonparatrading.com/en/python-trading-course/level-7/capstone-requirements-rubric-defense/), [5.2.4 · Leakage-free normalization and cross-sectional rank](https://pythonparatrading.com/en/python-trading-course/level-5/leakage-free-normalization-cross-sectional-rank/), [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/), [5.4.4 · Backtesting ML signals with costs and Deflated Sharpe](https://pythonparatrading.com/en/python-trading-course/level-5/backtesting-ml-signals-deflated-sharpe/), [5.5.6 · Learning to rank for stock selection](https://pythonparatrading.com/en/python-trading-course/level-5/learning-to-rank-stock-selection/), [4.1.4 · Risk parity, minimum variance, HRP and HERC](https://pythonparatrading.com/en/python-trading-course/level-4/risk-parity-minimum-variance-hrp-herc/), [7.4.6 · Trading taxes in Spain](https://pythonparatrading.com/en/python-trading-course/level-7/trading-taxes-in-spain/)

## By the end of this lesson

- Scope the brief of a stock and ETF portfolio managed with a ranking model, for an individual investor resident in Spain, and what is left out.
- Build the backbone with the level 5 panel: leakage audit, walk-forward ranking with LightGBM and a reference ridge, selection with a band, HRP and monthly rebalancing with costs.
- Keep FIFO lots with Spain's two-month rule and settle the IRPF on savings income every year to measure the after-tax return, reusing the logic of 7.4.6.
- Validate the choice of model and weights with a trial log and the Deflated Sharpe, and write the stopping criterion beforehand.
- Plan the paper trading so that it sees two rebalances and apply the common rubric of 7.6.1 to C3.

<aside class="callout" data-kind="riesgo">

**This is not tax or investment advice.** The project applies general rules of IRPF (Spain's personal income tax) for
residents of the common-regime territory (all of Spain except the Basque Country and Navarre; lesson 7.4.6) to a
simulated portfolio with synthetic data, with simplifications that are stated in the text. For a real tax return, the
source is the AEAT (Spain's tax agency) and, if needed, an adviser.

</aside>

## The brief

An individual investor resident in Spain wants a long-term stock portfolio managed with a model, and wants to know what
is left after costs and IRPF, not just what it earns in a backtest. The brief:

- **What is built.** A system that at the end of each month ranks a stock universe with a cross-sectional ranking
  model (LightGBM), buys the top 40 and holds them while they stay within the top 120, spreads risk with HRP, sends the
  orders to a paper account, keeps the tax lots with FIFO and Spain's two-month rule and settles the IRPF on savings
  income every year. Part of the portfolio can be a core of UCITS ETFs weighted with the same HRP (exercise 2).
- **For whom.** For the investor and for whoever reviews their tax return: a report with the gross return, the return
  net of costs and net of taxes, the turnover, the deferred losses and the statistical evidence, with the number of
  trials.
- **What is out.** Short positions, leverage, derivatives, US-domiciled ETFs (an EU retail investor cannot buy them
  without a KID, lesson 7.4.3), investment funds, dividends with foreign withholding (7.4.6 covers them) and any real
  money. Nor is it aggressive tax optimization: the project measures the effect of the rule and of an operating rule,
  it does not try to pay less.

Deliverables, in addition to the common ones of 7.6.1: the package with the data audit, the model and its reference,
the selection, HRP, the simulator with lots and the tax engine, the executor; `informe_c3.json` regenerated with one
command; the trial log; the paper journal with the reconciliation of positions and lots; and the technical report.

The guide refers by code to what has already been taught: cross-sectional normalization and ranking (5.2.4),
walk-forward with retraining (5.4.3), signal backtesting with costs and Deflated Sharpe (5.4.4), learning to rank and
its volatility trap (5.5.6), HRP (4.1.4), rebalancing with bands (4.1.7), taxation (7.4.6), PRIIPs and ETFs (7.4.3),
Interactive Brokers (4.4.2) and model monitoring (7.3.3).

## Architecture

```mermaid
flowchart TB
  D[Prices and features] --> A[Leakage audit]
  A --> M[Monthly ranking]
  M --> S[Selection and band]
  S --> H[HRP weights]
  H --> O[Orders and costs]
  O --> L[Lots and two months]
  L --> F[IRPF on savings]
  O --> C[Reconciliation]
  C --> V[IC monitor]
```

| Piece | What it does | Library | Where it was learned |
|---|---|---|---|
| Data audit | point-in-time, survivorship, leaks by anomalous IC | pandas, scipy | 2.5.1, 5.2.2 |
| Features | cross-sectional normal rank of each date | scipy | 5.2.4 |
| Model | LightGBM LambdaRank and regression; reference ridge | lightgbm, scikit-learn | 5.5.3, 5.5.6 |
| Validation | annual walk-forward with purging, log, DSR, PBO | numpy | 5.4.3, 5.4.4, 2.5.4 |
| Weights | hand-coded HRP (single linkage, bisection) | scipy | 4.1.4 |
| Rebalancing | entry and exit band, whole shares, costs | pandas | 4.1.7, 5.4.4 |
| Taxation | FIFO lots, two-month rule, loss offsetting, rate schedule | Python | 7.4.6 |
| Execution and paper | orders at the close, reconciliation of positions and lots | ib_async | 4.4.2, 4.5.5 |
| Monitor | live IC versus expected, drift, shutdown | MLflow, logs | 7.3.1, 7.3.3 |

HRP is coded by hand with scipy, as in lesson 4.1.4, for a practical reason and a methodological one. The practical
one: the project runs in the `ml` environment (LightGBM), where Riskfolio-Lib and skfolio are not installed. The
methodological one: lesson 4.1.4 checked that hand-written code and those libraries give the same weights with the same
configuration, and fixing the linkage and the leaf order in your own code avoids depending on a library's defaults.

## Data

The skeleton uses the synthetic level 5 panel (`public/curso/datos/nivel-5/`): 400 fictitious stocks from 2011 to
2025, with 22 weekly features known at the close of their date, forward labels and the generator's truth.

| File | What it contributes to the project |
|---|---|
| `panel5_features.parquet` | 22 point-in-time weekly features, one of them with a planted leak |
| `panel5_etiquetas.parquet` | return over the next week and over 21 sessions, with its closing date (for purging) |
| `panel5_precios.parquet` | daily total-return closes: returns, covariances and valuation |
| `panel5_verdad.parquet` | the true alpha of each stock and week: the ceiling for any model |
| `universo4_precios.parquet` (level 4) | 38 fictitious total-return indices by asset class, for the ETF core |

Three simplifications of the panel that the report has to state: there are no IPOs or delistings (survivorship bias
was covered with `universo2` in level 2), prices are total return (no separate dividends, so there is no income from
movable capital in IRPF) and there is no risk-free rate. The skeleton treats prices in USD as if they were euros; in a
real account, each purchase and each sale is converted at the exchange rate of its date (7.4.6).

For the real project, documented sources, without downloading anything in the cells:

- **SEC EDGAR**: the data.sec.gov APIs, with no authentication or keys according to the SEC itself, provide the XBRL
  facts of each company (`companyfacts`) and one fact for all companies and a period (`frames`)[^1]. They are the base
  for point-in-time fundamentals if the filing date of each report is saved. Automated access has to comply with its
  access policy; the report cites the version of that policy that was consulted.
- **Massive** (formerly Polygon.io): its pricing page offers, for individual use, a stocks plan with more than 20 years
  of history at 199 USD a month and a financials and ratios plan at 29 USD a month[^2]. The license is for individual
  use: it does not allow redistributing the data in the repository, so the repository keeps the download code and the
  fingerprints, not the data.
- **Kenneth R. French Data Library**: freely downloadable factors and benchmark portfolios, for the portfolio
  attribution (lesson 4.1.6).
- **UCITS ETFs**: the KID and the issuer's factsheet for each ETF in the core (lesson 7.4.3), with its TER, its
  dividend policy (accumulating or distributing) and its currency.

The central problem with real stock data is the universe: a universe built with the companies listed today has
survivorship bias, and one with fundamentals without a publication date has look-ahead. The project's data README
documents how the universe is fixed at each date and where the date of each data point comes from.

## Runnable skeleton

Six cells, from the features to the report. The first builds the monthly panel (the last panel date of each month, the
rebalancing day) and, before training anything, audits the features: the monthly IC of each one with the 21-session
return, only in the first training window (2012-2014), so that the filter does not look at the future either.

```python
import numpy as np
import pandas as pd
from scipy.stats import norm, rankdata

feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet",
                       columns=["fecha", "ticker", "ret_sig_21d",
                                "fin_sig_21d"])
d = feat.merge(etiq, on=["fecha", "ticker"])
# the last panel date of each calendar month: rebalancing day
fin_mes = d.groupby(d["fecha"].dt.to_period("M"))["fecha"].max()
d = d[d["fecha"].isin(fin_mes)].sort_values(["fecha", "ticker"])
TODAS = list(feat.columns[3:])
fechas = pd.DatetimeIndex(d["fecha"].unique())
nM, nA = len(fechas), d["ticker"].nunique()
tickers = d["ticker"].unique()[:nA]
F = d[TODAS].to_numpy(float).reshape(nM, nA, -1)
Y = d["ret_sig_21d"].to_numpy().reshape(nM, nA)
fin = d["fin_sig_21d"].to_numpy().reshape(nM, nA)[:, 0]


def rango_normal(X):
    """Cross-sectional normal rank of each month (5.2.4); no data, 0."""
    n = np.isfinite(X).sum(axis=1, keepdims=True)
    with np.errstate(invalid="ignore", divide="ignore"):
        u = (rankdata(X, axis=1, nan_policy="omit") - 0.5) / n
        return np.nan_to_num(norm.ppf(u))


G = np.stack([rango_normal(F[:, :, j]) for j in range(len(TODAS))], 2)
# audit: monthly IC of each feature in 2012-2014 (only the first
# training window) and its t; a huge t is an alarm
tramo = (fechas.year < 2015) & np.isfinite(Y).all(axis=1)
RY = rango_normal(Y)
ic = np.array([[np.corrcoef(G[m, :, j], RY[m])[0, 1]
                for m in np.flatnonzero(tramo)]
               for j in range(len(TODAS))])
t = ic.mean(1) / ic.std(1, ddof=1) * np.sqrt(ic.shape[1])
auditoria = pd.DataFrame({"ic": ic.mean(1), "t": t}, index=TODAS)
print(f"{nM} months x {nA} stocks; audited window: "
      f"{tramo.sum()} months")
print(auditoria.sort_values("t", key=abs, ascending=False)
      .head(4).round(3).to_string())
SOSPECHOSAS = auditoria.index[auditoria["t"].abs() > 5].tolist()
FE = [j for j, c in enumerate(TODAS) if c not in SOSPECHOSAS]
print("out:", SOSPECHOSAS, "| remaining", len(FE), "features")
```
```output
168 months x 400 stocks; audited window: 36 months
                       ic       t
revision_analistas  0.096  10.815
mom_12_1            0.038   2.765
sue                 0.023   2.301
dist_max_52s        0.027   1.979
out: ['revision_analistas'] | remaining 21 features
```

One hundred sixty-eight months, 400 stocks. `revision_analistas` (the analyst revision) has an IC of 0.096 with
t = 10.8 over 36 months; the next one, momentum, 0.038 with t = 2.8. A feature that ranks two and a half times better
than momentum in a stock panel is an alarm, and here the README confirms why: the vendor dates it at the start of the
week it covers and it contains later returns. The threshold (|t| > 5) is an audit rule written beforehand, not a
statistical test; whatever fails it is reviewed by hand and documented.

The model is retrained every January with all the months whose label had already closed before the first rebalance of
the year (the purging of 5.4.3: the 21-session labels of the last month overlap with the first test month). Three
models with the same features in normal rank: a reference ridge, LightGBM as a regressor of the return rank and
LightGBM LambdaRank with quintiles as grades, linear gain and truncation at 40, the number of stocks bought (the
recommendations of 5.5.6). Moderate, fixed hyperparameters, without a search; the seed and the threads, fixed.

```python
import lightgbm as lgb
from sklearn.linear_model import Ridge

ok = np.isfinite(Y).all(axis=1)
RY = np.where(ok[:, None], rango_normal(Y), np.nan)  # label
quintil = np.where(ok[:, None], (rankdata(Y, axis=1) - 1) * 5 // nA,
                   -1).astype(int)
ARB = dict(n_estimators=200, learning_rate=0.03, num_leaves=15,
           min_child_samples=200, subsample=0.7, subsample_freq=1,
           colsample_bytree=0.7, random_state=0, n_jobs=4,
           deterministic=True, force_row_wise=True, verbose=-1)
anyo = fechas.year.to_numpy()
punt = {k: np.full((nM, nA), np.nan)
        for k in ("ridge", "lgbm_reg", "lgbm_rank")}
for y in range(2015, 2026):
    hoy = np.flatnonzero(anyo == y)
    # time rule: labels closed before the first rebalance of the
    # year (purging the overlapping months)
    tr = np.flatnonzero(ok & (fin < fechas[hoy[0]]))
    X = G[tr][:, :, FE].reshape(-1, len(FE))
    mod = {"ridge": Ridge(alpha=10.0).fit(X, RY[tr].ravel()),
           "lgbm_reg": lgb.LGBMRegressor(**ARB).fit(X, RY[tr].ravel()),
           "lgbm_rank": lgb.LGBMRanker(
               objective="lambdarank", label_gain=[0, 1, 2, 3, 4],
               lambdarank_truncation_level=40, **ARB).fit(
               X, quintil[tr].ravel(), group=[nA] * len(tr))}
    for k, m in mod.items():
        punt[k][hoy] = m.predict(G[hoy][:, :, FE].reshape(
            -1, len(FE))).reshape(len(hoy), nA)
oos = np.flatnonzero((anyo >= 2015) & ok)
verdad = pd.read_parquet("panel5_verdad.parquet",
                         columns=["fecha", "ticker", "alfa_verdadero"])
verdad = verdad[verdad["fecha"].isin(fechas)].sort_values(
    ["fecha", "ticker"])
punt["oraculo"] = verdad["alfa_verdadero"].to_numpy().reshape(nM, nA)
vol = F[:, :, TODAS.index("vol_63d")]


def corr_r(a, b):
    return np.mean([np.corrcoef(rankdata(a[i]), rankdata(b[i]))[0, 1]
                    for i in oos])


filas = {}
for k, p in punt.items():
    x = np.array([np.corrcoef(rankdata(p[i]), rankdata(Y[i]))[0, 1]
                  for i in oos])
    filas[k] = {"IC": x.mean(),
                "t": x.mean() / x.std(ddof=1) * np.sqrt(len(x)),
                "corr_vol": corr_r(p, vol)}
print(f"{len(oos)} out-of-sample months (2015-2025)")
pd.DataFrame(filas).T.round(4)
```
```output
131 out-of-sample months (2015-2025)
               IC       t  corr_vol
ridge      0.0221  3.1468    0.0999
lgbm_reg   0.0124  2.1227    0.0859
lgbm_rank  0.0096  1.2893    0.2815
oraculo    0.0276  3.3056   -0.2739
```

Over 131 out-of-sample months, the ridge has an IC of 0.0221 (t = 3.15), close to the oracle (`oraculo`, 0.0276;
t = 3.31), which knows the true alpha. LightGBM as a regressor gives 0.0124 and as LambdaRank, 0.0096, with t = 1.29
and a rank correlation of 0.28 with `vol_63d`: the ranker once again loads up on volatility, as in lesson 5.5.6. The
oracle has a negative correlation with volatility (−0.27) because the panel plants a low-volatility premium. The fact
that the brief asks for LightGBM does not change the method: the model is presented alongside its reference.

The portfolio buys the top 40 and keeps the ones it already holds while they stay within the top 120 (the 10% and 30%
band of 5.4.4, fixed beforehand), with equal weights (`igual`) or with HRP on the covariance of the 252 sessions before
the rebalance. It pays 10 bp per side on turnover. The six combinations are the trial log; the reference is the
equal-weighted universe, and the active return, the monthly difference from it.

```python
from scipy.cluster.hierarchy import leaves_list, linkage
from scipy.spatial.distance import squareform

px = pd.read_parquet("panel5_precios.parquet",
                     columns=["fecha", "ticker", "cierre"])
px = px.pivot(index="fecha", columns="ticker", values="cierre")
px = px[tickers].astype(float)
ret = px.pct_change()
TOP, COSTE = 40, 0.001            # 40 stocks; 10 bp per side


def var_ivp(S, ix):
    c = S[np.ix_(ix, ix)]
    w = 1 / np.diag(c)
    w /= w.sum()
    return w @ c @ w


def hrp(S):
    """HRP from lesson 4.1.4 (single linkage, bisection)."""
    sd = np.sqrt(np.diag(S))
    dist = np.sqrt(np.clip((1 - S / np.outer(sd, sd)) / 2, 0, 1))
    orden = leaves_list(linkage(squareform(dist, checks=False),
                                "single"))
    w, grupos = np.ones(len(S)), [orden]
    while grupos:
        sig = []
        for g in grupos:
            if len(g) > 1:
                a, b = g[: len(g) // 2], g[len(g) // 2:]
                va, vb = var_ivp(S, a), var_ivp(S, b)
                w[a] *= vb / (va + vb)
                w[b] *= va / (va + vb)
                sig += [a, b]
        grupos = sig
    return w


REB = [i for i in range(nM) if anyo[i] >= 2015]
BANDA = 120                       # held while within the top 120


def seleccion(p, i, tenidas, vetadas=()):
    """The TOP best, keeping the ones already held while they stay
    within the top BANDA (5.4.4); `vetadas` do not come in."""
    orden = [j for j in np.argsort(-p[i]) if tickers[j] not in vetadas]
    rango = {j: n for n, j in enumerate(orden)}
    sel = sorted((j for j in tenidas if rango.get(j, nA) < BANDA),
                 key=rango.get)[:TOP]
    sel += [j for j in orden if j not in sel][:TOP - len(sel)]
    return np.array(sel)


def pesos(sel, i, metodo):
    w = np.zeros(nA)
    if metodo == "igual":
        w[sel] = 1 / len(sel)
    else:
        hist = ret.loc[:fechas[i]].iloc[-252:].to_numpy()[:, sel]
        w[sel] = hrp(np.cov(hist, rowvar=False))
    return w


def cartera(p, metodo):
    """Monthly return net of costs, and turnover."""
    w_ant, filas = np.zeros(nA), []
    for a, b in zip(REB[:-1], REB[1:]):
        w = pesos(seleccion(p, a, np.flatnonzero(w_ant)), a, metodo)
        rot = np.abs(w - w_ant).sum()
        r = (px.loc[fechas[b]] / px.loc[fechas[a]] - 1).to_numpy()
        bruto = float(w @ r)
        filas.append((fechas[b], bruto - COSTE * rot, rot))
        w_ant = w * (1 + r) / (1 + bruto)
    return pd.DataFrame(filas, columns=["fecha", "neto", "rot"]
                        ).set_index("fecha")


mercado = pd.Series(
    [(px.loc[fechas[b]] / px.loc[fechas[a]] - 1).mean()
     for a, b in zip(REB[:-1], REB[1:])],
    index=fechas[REB[1:]])                 # equal-weighted universe
pruebas = {f"{k} + {m}": cartera(punt[k], m)
           for k in ("ridge", "lgbm_reg", "lgbm_rank")
           for m in ("igual", "hrp")}
activo = pd.DataFrame({k: v["neto"] - mercado
                       for k, v in pruebas.items()})
registro = pd.DataFrame({
    "activo_anual_%": 100 * 12 * activo.mean(),
    "ir": activo.mean() / activo.std() * np.sqrt(12),
    "sharpe": [v["neto"].mean() / v["neto"].std() * np.sqrt(12)
               for v in pruebas.values()],
    "rotacion_mes_%": [100 * v["rot"].iloc[1:].mean()
                       for v in pruebas.values()]})
registro.to_csv("registro_pruebas.csv")
techo = {m: 12 * (cartera(punt["oraculo"], m)["neto"]
                  - mercado).mean() for m in ("igual", "hrp")}
print(f"{len(mercado)} months; universe: "
      f"{100 * ((1 + mercado).prod() ** (12 / len(mercado)) - 1):.1f}"
      f" % a year, Sharpe "
      f"{mercado.mean() / mercado.std() * np.sqrt(12):.2f}")
print("ceiling (oracle), annual active return:",
      {k: f"{100 * v:.1f} %" for k, v in techo.items()})
registro.round(2)
```
```output
131 months; universe: 6.5 % a year, Sharpe 0.43
ceiling (oracle), annual active return: {'igual': '2.0 %', 'hrp': '0.0 %'}
                   activo_anual_%    ir  sharpe  rotacion_mes_%
ridge + igual                1.97  0.41    0.49           45.42
ridge + hrp                  0.81  0.15    0.48           57.87
lgbm_reg + igual             0.63  0.14    0.42           82.24
lgbm_reg + hrp              -0.52 -0.10    0.40           91.31
lgbm_rank + igual            0.73  0.13    0.43           76.36
lgbm_rank + hrp             -0.06 -0.01    0.44           89.16
```

*Fig. — Capstone C3: the 6 trials in the log (model and weights), ranked by the information ratio of their active return net of costs against the equal-weighted universe, with the maximum expected by luck.* Six dots on an annual IR axis from −0.3 to 0.6. Ridge with equal weights is on top, at 0.41; the others between 0.15 and −0.10. The highlighted one is the brief's configuration, LambdaRank with HRP, at −0.01. A dashed line marks 0.23, the maximum IR expected by luck with 6 trials: only ridge with equal weights exceeds it, with a Deflated Sharpe of 0.727.

The universe rises 6.5% a year with a Sharpe of 0.43 (no risk-free rate). The best trial, ridge with equal weights,
beats it by 1.97% a year with an IR of 0.41 and the lowest turnover (45% a month); the one of the brief, LambdaRank
with HRP, stays at −0.06% with an IR of −0.01 and a turnover of 89%. The figure ranks the six by IR. The absolute
Sharpes are all between 0.40 and 0.49: HRP does not change the Sharpe much, but it does change the active return. The
known answer explains it: the oracle with equal weights beats the universe by 2.0% and with HRP, by 0.0%. HRP weights
by risk, loads the least volatile stocks and, against an equal-weighted universe in a rising market, eats up what the
selection contributes. It is a risk control, not a source of return.

The tax part reuses the logic of the 7.4.6 engine in a compact floating-point version, designed to simulate eleven
years of rebalances and not to prepare the tax return (which uses exact decimals, as in 7.4.6). The lots follow FIFO
(art. 37.2 of the IRPF law) and the two-month rule (art. 33.5 f): purchases made within two months after a sale at a
loss defer the loss of the repurchased units, and earlier purchases only count up to the remaining balance. The
deferred loss travels with the repurchased lot and is recognized when it is sold. A test with a case computed by hand:

```python
from collections import defaultdict, deque
from dateutil.relativedelta import relativedelta

DOS_MESES = relativedelta(months=2)


class Lotes:
    """FIFO lots by value (art. 37.2) with the two-month rule
    (art. 33.5 f, lesson 7.4.6), simplified: see the text."""

    def __init__(self):
        # lot: [date, units, cost, deferred, free units]
        self.cola = defaultdict(deque)
        self.pend = defaultdict(list)    # loss sales still to match
        self.lineas = []                 # (year, amount, type)
        self.encadenadas = 0

    def comprar(self, f, t, uds, coste):
        dif, libres = 0.0, uds
        for v in self.pend[t]:           # purchase after a sale
            if v["fecha"] + DOS_MESES >= f and v["uds"] > 0 and libres:
                u = min(v["uds"], libres)
                dif += u * v["perdida_ud"]
                self.encadenadas += v["encadenada"]
                self.lineas.append((v["fecha"].year,
                                    u * v["perdida_ud"], "difiere"))
                v["uds"] -= u
                libres -= u
        self.cola[t].append([f, uds, coste, dif, libres])

    def vender(self, f, t, uds, importe):
        cola, falta, coste, liberada = self.cola[t], uds, 0.0, 0.0
        while falta:
            lote = cola[0]
            u = min(falta, lote[1])
            x = u / lote[1]
            coste += lote[2] * x
            liberada += lote[3] * x
            lote[1] -= u
            lote[2] -= lote[2] * x
            lote[3] -= lote[3] * x
            lote[4] = min(lote[4], lote[1])
            falta -= u
            if lote[1] == 0:
                cola.popleft()
        res = importe - coste
        self.lineas.append((f.year, res, "venta"))
        if liberada:
            self.lineas.append((f.year, -liberada, "integra"))
            res -= liberada
        if res >= 0:
            return
        quedan = sum(l[1] for l in cola)
        n = 0                            # earlier purchases, up to
        for lote in reversed(cola):      # the remaining balance
            if lote[0] + DOS_MESES >= f and quedan:
                u = min(lote[4], quedan - n)
                lote[3] += -res / uds * u
                lote[4] -= u
                n += u
        if n:
            self.lineas.append((f.year, -res / uds * n, "difiere"))
            self.encadenadas += liberada > 0
        if uds > n:
            self.pend[t].append({"fecha": f, "uds": uds - n,
                                 "perdida_ud": -res / uds,
                                 "encadenada": liberada > 0})


# test: buy 100, sell 100 at a loss, buy back 60 a month later
L = Lotes()
L.comprar(pd.Timestamp("2025-01-10"), "ABCD", 100, 10_000.0)
L.vender(pd.Timestamp("2025-03-10"), "ABCD", 100, 8_000.0)
L.comprar(pd.Timestamp("2025-04-08"), "ABCD", 60, 4_900.0)
L.vender(pd.Timestamp("2025-09-15"), "ABCD", 60, 5_400.0)
for linea in L.lineas:
    print(linea)
print("year 2025:", round(sum(i for a, i, _ in L.lineas
                              if a == 2025), 2))
```
```output
(2025, -2000.0, 'venta')
(2025, 1200.0, 'difiere')
(2025, 500.0, 'venta')
(2025, -1200.0, 'integra')
year 2025: -1500.0
```

One hundred shares bought for €10,000 and sold for €8,000: a loss of €2,000. Sixty are bought back a month later:
60 × 20 = €1,200 is deferred. When those 60 are sold with a gain of €500, the deferred loss is recognized: the year
adds up to −€1,500, the total economic loss. The line types in the output are the engine's identifiers: `venta`
(sale), `difiere` (deferral) and `integra` (deferred loss recognized). Two simplifications relative to 7.4.6, which the
report has to declare: the case of a single prior purchase with no shares held at the start of the period is only
handled through the limit of the remaining balance, and when shares with a deferred loss are sold and there are
repurchases again (the case that 7.4.6 says to review by hand), the simulator defers the total loss of that sale
again.

The simulator keeps the portfolio in whole shares, sells before buying, pays the costs and, at the first July
rebalance, the previous year's IRPF (when the two-month windows of the December sales have already closed), with the
2026 savings rate schedule and the losses of the previous four years. At the end it sells everything and settles the
last year, to compare net values. Three variants of the portfolio of the brief: without IRPF, with IRPF and with IRPF
plus a two-month lockout on buying what was sold at a loss (the “repurchase warning” from the 7.4.6 exercise).

```python
TRAMOS = [(6000, 0.19), (50000, 0.21), (200000, 0.23),
          (300000, 0.27), (np.inf, 0.30)]      # 2026 schedule


def cuota(base):
    c, desde = 0.0, 0.0
    for hasta, t in TRAMOS:
        c += max(0.0, min(base, hasta) - desde) * t
        desde = hasta
    return c


def impuesto(lineas, y, pend):
    """Tax due for the year on gains and losses (no dividends: prices
    are total return), with the losses of the previous four years
    (art. 49). Modifies `pend`."""
    saldo = sum(i for a, i, _ in lineas if a == y)
    for a in sorted(pend):
        if a < y - 4:
            del pend[a]
        elif saldo > 0:
            x = min(pend[a], saldo)
            pend[a] -= x
            saldo -= x
    if saldo < 0:
        pend[y] = -saldo
    return cuota(max(saldo, 0.0))


def simular(p, metodo, fiscal=True, bloqueo=False, capital=100_000.0):
    """Portfolio in whole shares: sells, buys, pays 10 bp per side
    and, at the first July rebalance, the previous year's IRPF. At
    the end it sells everything and settles the last year.
    `bloqueo`: what was sold at a loss is not bought or increased for
    two months (repurchase warning of 7.4.6)."""
    caja, uds, L, pend = capital, defaultdict(int), Lotes(), {}
    vetos, pagado, valor = {}, {}, {}
    for i in REB:
        f, precio = fechas[i], px.loc[fechas[i]]
        y = f.year - 1
        if fiscal and f.month >= 7 and y >= 2015 and y not in pagado:
            pagado[y] = impuesto(L.lineas, y, pend)
            caja -= pagado[y]
        valor[f] = caja + sum(n * precio[t] for t, n in uds.items())
        vet = {t for t, h in vetos.items() if h > f and bloqueo}
        if i == REB[-1]:
            obj = {}                     # final liquidation
        else:
            tenidas = [j for j, t in enumerate(tickers) if uds[t] > 0]
            fuera = [t for t in vet if uds[t] == 0]
            sel = seleccion(p, i, tenidas, fuera)
            w = pesos(sel, i, metodo)
            obj = {tickers[j]: int(w[j] * valor[f] * 0.995
                                   / precio.iloc[j]) for j in sel}
        for t in [t for t in uds if uds[t] > obj.get(t, 0)]:
            n = uds[t] - obj.get(t, 0)
            imp = n * precio[t] * (1 - COSTE)
            k = len(L.lineas)
            L.vender(f, t, n, imp)
            if any(x[2] == "difiere" for x in L.lineas[k:]) or (
                    L.pend[t] and L.pend[t][-1]["fecha"] == f):
                vetos[t] = f + DOS_MESES + pd.Timedelta(days=1)
            caja += imp
            uds[t] -= n
        for t, n_obj in obj.items():
            n = n_obj - uds[t]
            if n > 0 and t not in vet:
                coste = n * precio[t] * (1 + COSTE)
                L.comprar(f, t, n, coste)
                caja -= coste
                uds[t] += n
    if fiscal:
        pagado[f.year] = impuesto(L.lineas, f.year, pend)
        caja -= pagado[f.year]
    return pd.Series(valor), caja, L, pagado


def lineas_anuales(L, regla=True):
    """Balance of gains and losses per year; without the rule, like
    an engine that defers nothing."""
    tipos = ("venta", "difiere", "integra") if regla else ("venta",)
    return pd.Series([i for _, i, k in L.lineas if k in tipos],
                     index=[a for a, _, k in L.lineas if k in tipos]
                     ).groupby(level=0).sum()


t_res = {}
for nombre, kw in (("no IRPF", dict(fiscal=False)),
                   ("with IRPF", {}),
                   ("with IRPF and lockout", dict(bloqueo=True))):
    v, neto, L, pagado = simular(punt["lgbm_rank"], "hrp", **kw)
    anos = (v.index[-1] - v.index[0]).days / 365.25
    perd = -sum(i for _, i, k in L.lineas if k == "venta" and i < 0)
    t_res[nombre] = {
        "final_neto": neto,
        "tae_%": 100 * ((neto / v.iloc[0]) ** (1 / anos) - 1),
        "irpf": sum(pagado.values()),
        "perdidas": perd,
        "diferidas_%": 100 * sum(i for _, i, k in L.lineas
                                 if k == "difiere") / perd}
    if nombre == "with IRPF":
        L_ref, v_ref = L, v
print(pd.DataFrame(t_res).T.round(1).to_string())
con, sin = lineas_anuales(L_ref), lineas_anuales(L_ref, False)
print("Year's balance with the rule minus without it (€):")
print(pd.DataFrame({"": (con - sin).round(0).astype(int)}).T
      .to_string())
```
```output
                       final_neto  tae_%     irpf  perdidas  diferidas_%
no IRPF                  198790.8    6.5      0.0  285786.7         46.6
with IRPF                173348.7    5.2  18766.4  275682.5         46.5
with IRPF and lockout    164072.1    4.6  16510.8  262669.1         29.8
Year's balance with the rule minus without it (€):
  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  2025
  5414 -2677  -545  4684 -4838   934  -232   799 -1903  1039 -2677
```

Starting from €100,000 in January 2015, the portfolio without IRPF ends at €198,791 (annualized return, `tae_%`, of
6.5%); with IRPF, at €173,349 (5.2%), after paying €18,766 of tax over eleven years. The header figure compares the
three with the universe. In the variant with IRPF, €275,683 of gross losses are realized over eleven years, and 46.5%
are deferred at some point by the two-month rule: the band keeps the stocks that remain well ranked, but HRP trims and
increases positions every month, and a partial sale at a loss after a purchase the previous month falls inside the
window. (In the row without IRPF the last two columns only describe the lots; nothing is paid.)

The last table is the error of an engine that does not model the rule: the difference, year by year, between the
balance of gains and losses with the rule and without it. In 2015 the rule postpones €5,414 of losses that a naive
engine would have declared that year; in 2025, when everything is liquidated, €2,677 that had been deferred are
recognized. The row sums to zero (except for rounding): the rule moves taxable base between years, it does not change
the total. The lockout reduces the deferrals to 29.8% and the tax paid, but the net annualized return drops to 4.6%: by
vetoing well-ranked stocks for two months, the portfolio moves away from the one the model asks for, and on this data
that costs more than the tax timing it saves.

The last cell discounts the selection with the Deflated Sharpe on the monthly IR of the six trials and writes the
artifact, `informe_c3.json`, with the data fingerprint, what the audit set aside, the validation and the tax summary.

```python
import hashlib
import json


def psr(sr, umbral, t, asim, curt):
    den = np.sqrt(1 - asim * sr + (curt - 1) / 4 * sr**2)
    return norm.cdf((sr - umbral) * np.sqrt(t - 1) / den)


def max_esperado(n):
    gm = np.euler_gamma
    return ((1 - gm) * norm.ppf(1 - 1 / n)
            + gm * norm.ppf(1 - 1 / (n * np.e)))


ir = activo.mean() / activo.std()          # per month
mejor = ir.idxmax()
x = activo[mejor]
sr0 = ir.std() * max_esperado(len(ir))
mom = (len(x), x.skew(), x.kurt() + 3)
dsr = psr(ir[mejor], sr0, *mom)
print(f"best of {len(ir)}: {mejor}, IR {ir[mejor] * np.sqrt(12):.2f};"
      f" threshold {sr0 * np.sqrt(12):.2f}; DSR {dsr:.3f}")
pre = "lgbm_rank + hrp"                     # the one of the brief
xp = activo[pre]
print(f"brief ({pre}): IR {ir[pre] * np.sqrt(12):.2f}, PSR(0) "
      f"{psr(ir[pre], 0, len(xp), xp.skew(), xp.kurt() + 3):.3f}")


def sha(ruta):
    return hashlib.sha256(open(ruta, "rb").read()).hexdigest()[:12]


informe = {
    "proyecto": "C3 ML portfolio, HRP and IRPF", "config": pre,
    "datos": {a: sha(a) for a in ("panel5_features.parquet",
                                  "panel5_precios.parquet")},
    "fuera": SOSPECHOSAS,
    "validacion": {"n_pruebas": len(ir), "mejor": mejor,
                   "dsr_mejor": round(float(dsr), 3),
                   "ir_encargo": round(float(ir[pre]) * 12**0.5, 3)},
    "fiscal": {k: {c: round(float(v), 1) for c, v in d.items()}
               for k, d in t_res.items()}}
with open("informe_c3.json", "w") as fh:
    json.dump(informe, fh, ensure_ascii=False, indent=1)
print(json.dumps(informe["validacion"], ensure_ascii=False))
```
```output
best of 6: ridge + igual, IR 0.41; threshold 0.23; DSR 0.727
brief (lgbm_rank + hrp): IR -0.01, PSR(0) 0.487
{"n_pruebas": 6, "mejor": "ridge + igual", "dsr_mejor": 0.727, "ir_encargo": -0.01}
```

## Validation

In C3 four things are chosen, and each one has its control:

| What is chosen | Risk | Control in the project |
|---|---|---|
| Features | a leak that looks like the best feature | IC audit on the first window; list of excluded features in the report |
| Model | comparing against nothing or tuning until it wins | reference ridge; fixed hyperparameters; log |
| Portfolio (weights, band, size) | choosing the best combination in the same sample | trial log, DSR, PBO |
| Period and benchmark | choosing the window or index that favors it | 2015-2025 fixed in week 1; equal-weighted universe declared |

The walk-forward is built into the construction itself: each year it trains with what was known on December 31, with
the overlapping labels purged, and predicts the next twelve months without touching the model. On top of that
walk-forward, the log has six trials; the best, ridge with equal weights, has an IR of 0.41 and a DSR of 0.727 with
N = 6: the threshold demanded by the dispersion of the six (0.23 of annual IR) eats up a good part of the evidence. The
one of the brief has a PSR(0) of 0.487: nothing distinguishable from zero. Exercise 1 adds the probability of backtest
overfitting by CSCV.

The **stopping criterion**, written in week 4 before computing:

- If the audit finds a leak that cannot be explained and removed, the project stops until it is resolved.
- If the out-of-sample IC of the model of the brief is not positive with t > 2, or its active return net of costs is
  not positive, the portfolio is not presented as active management: the paper trading is done to evaluate the
  system, and the report compares it with the passive alternative (the universe or an ETF core).
- If the DSR of the best trial is < 0.95, the report says that there is no statistical evidence of an edge after the
  trials run.
- With the course data, the model of the brief does not pass (IC with t = 1.29, IR of −0.01) and the best trial does
  not reach the DSR either. The system is evaluated on paper; the report's conclusion is that, with this panel and
  monthly rebalancing, the selection does not make up for costs and taxes against the universe with enough evidence.

The skeleton not finding an edge is not a failure of the project: the panel has a small planted alpha, which lives
mostly for a week, and long-term monthly rebalancing only captures part of it. The known answer (the oracle beats the
universe by 2.0% a year with equal weights) marks the ceiling. A project that, with this data, presented an IR of 1.5
would have to explain where it comes from before the committee asks about the leak.

## Risks and paper trading

Risks that really bring down a portfolio of this kind:

- **Data.** Survivorship, fundamentals without a publication date, wrongly applied corporate actions and a vendor's
  leak. A single feature with look-ahead invalidates everything; that is why the audit comes before the model and is
  repeated with every version of the data.
- **A model that stops working.** The panel's IC is small and changes with regimes (the momentum crash, 5.4.3); the
  monitor of 7.3.3 compares the live IC with the expected one and shuts the model down with its graduated policy.
- **Concentration and factors.** 40 stocks chosen by a model can be a sector or volatility bet; limits per sector and
  factor attribution (4.1.6) watch for it.
- **Costs and taxation.** The turnover of a monthly model is high (89% a month in the portfolio of the brief); each sale
  with a gain brings tax forward. The report always gives the three figures: gross, net of costs and net of taxes.
- **Operational.** Orders that are not filled at the close, fractional shares, currency conversions and lots that do
  not match the broker's report.

Limits in the configuration: maximum weight per stock and per sector, maximum turnover per rebalance, minimum cash, and
a shutdown if the live IC falls below what is expected (7.3.3) or if the reconciliation does not match.

```mermaid
stateDiagram-v2
  [*] --> AUDITING
  AUDITING --> SCORING
  AUDITING --> BLOCKED
  SCORING --> ORDERING
  ORDERING --> RECONCILING
  RECONCILING --> WAITING
  RECONCILING --> BLOCKED
  WAITING --> AUDITING
  BLOCKED --> AUDITING
```

**Paper trading plan.** With one rebalance a month, four weeks from any given day would see only one and no sale of
what was already held. Paper starts a few days before a month-end and runs until after the next one: at least four
weeks (28 days and 20 sessions, 7.6.1) and two rebalances with real paper orders. Between the two, the system
reconciles positions and cash with the broker every day, watches weight drift and corporate actions, and logs the IC of
the week. After the second rebalance, your own lot record is squared with the broker's trade report: if the broker lets
you choose the lot assignment method, your own tax report follows FIFO (art. 37.2), whatever the broker's method is.
The executor sends orders at the close, which is the price the backtest uses; this cell is the real code and is not
run here:

```python skip
from ib_async import IB, Order, Stock

ib = IB()
ib.connect("127.0.0.1", 4002, clientId=34)       # IB Gateway, paper
cartera = {p.contract.symbol: p.position for p in ib.positions()}
objetivo = calcular_objetivo(cartera)              # signal + HRP + band
ventas = {t: n - objetivo.get(t, 0) for t, n in cartera.items()
          if n > objetivo.get(t, 0)}
compras = {t: n - cartera.get(t, 0) for t, n in objetivo.items()
           if n > cartera.get(t, 0)}
for lado, lista in (("SELL", ventas), ("BUY", compras)):
    for t, n in sorted(lista.items()):
        c = ib.qualifyContracts(Stock(t, "SMART", "USD"))[0]
        orden = Order(action=lado, totalQuantity=n, orderType="MOC",
                      orderRef=f"C3-{t}-{lado}-20261030")
        ib.placeOrder(c, orden)                    # at the close
ib.sleep(5)
print([(t.contract.symbol, t.orderStatus.status) for t in ib.trades()])
```

Three details for the real executor: sales are sent before purchases and purchases are recomputed with the cash
actually available if a sale is not filled; the key of each order (`orderRef`) comes from the date, the stock and the
side (4.5.5); and a stock vetoed by the repurchase lockout, if used, is recorded with its end date in the system's own
state, not in the process memory.

## Milestones and rubric

The schedule is the one from 7.6.1 (ten weeks, 40 to 80 hours), applied to C3:

| Week | Verifiable deliverable |
|---|---|
| 1 | Repository from the template; data README with sources, licenses, universe by date and fingerprints; 7.6.1 auditor green |
| 2 | Point-in-time loading, normal rank, leakage audit with its report; data tests |
| 3 | Model and reference in walk-forward with purging; selection with a band, HRP, costs; lot engine tested with the cases of 7.4.6 |
| 4 | Trial log, DSR and PBO; stopping criterion applied; tax simulation; `informe_c3.json` |
| 5 | Paper executor with `ib_async` against a simulated broker and the paper account; Docker; IC monitor |
| 6-10 | Paper from a few days before a month-end until after the next one (about five weeks), with journal and reconciliation of positions and lots; report in parallel |
| 10 | Paper report after closing the second rebalance; rehearsal and defense |

The common rubric of 7.6.1, with the specific evidence it asks for in C3:

| Criterion (weight) | Evidence in C3 |
|---|---|
| Reproducibility (15%) | `make reproduce` regenerates `informe_c3.json` with the report's fingerprint; fixed LightGBM seeds and threads; uv.lock, Docker |
| Data (15%) | universe by date without survivorship; publication date of each fundamental; license of each source; leakage audit |
| Validation (20%) | walk-forward with purging; linear reference; complete log; DSR and PBO; stopping criterion written beforehand |
| Risk and execution (15%) | costs per side; whole shares; weight, sector and turnover limits; IRPF by lots with the two-month rule |
| Paper trading (15%) | at least 28 days and two rebalances; daily reconciliation; lots squared with the broker; weekly IC logged |
| Code (10%) | tests of the audit, of HRP (weights summing to 1, known cases), of the lot engine against hand-computed cases; ruff and pre-commit |
| Report and defense (10%) | gross, net and after-tax return; tax simplifications declared; limits of the study |

Mistakes that fail on their own in this project, in addition to the common ones: training with `revision_analistas`
or with any feature dated after the decision; normalizing with statistics from the whole sample (5.2.4); an after-tax
return computed with a flat rate on the year's gain, without lots or a rate schedule (that is not the IRPF on savings
income); and presenting the model of the brief without the linear reference over the same period.

## Exercises

1. Compute the probability of backtest overfitting (PBO) of the choice among the six trials with CSCV: eight time
   blocks of the active return, all the partitions into four training blocks and four test blocks, and the
   out-of-sample position of the best in-sample trial (lesson 2.5.4).

**Solution.**

Over 70 partitions, the trial chosen in sample falls below the out-of-sample median in 30% of the cases: a PBO of 0.30.
It is not that of a random choice (0.5), but it does not rule out that the winner wins because of the period either.
Together with the DSR of 0.727, the conclusion is the same: with six trials and eleven years, the evidence that one
combination is better than the others is weak.

```python
from itertools import combinations

M = activo.to_numpy()
bloques = np.array_split(np.arange(len(M)), 8)
logits = []
for tr in combinations(range(8), 4):
    i_tr = np.concatenate([bloques[b] for b in tr])
    i_te = np.concatenate([bloques[b] for b in range(8)
                           if b not in tr])
    ir_tr = M[i_tr].mean(0) / M[i_tr].std(0)
    ir_te = M[i_te].mean(0) / M[i_te].std(0)
    k = int(np.argmax(ir_tr))
    w = (rankdata(ir_te)[k]) / (len(ir_te) + 1)
    logits.append(np.log(w / (1 - w)))
logits = np.array(logits)
print(f"{len(logits)} partitions; PBO = {(logits <= 0).mean():.2f}")
```
```output
70 partitions; PBO = 0.30
```

2. Add an ETF core: monthly HRP on seven blocks of the level 4 universe that would have a UCITS ETF (US, European,
   Japanese and emerging-market equities, hedged global bonds, 3-7 year US Treasuries and global real estate), and
   combine it 60/40 with the stock portfolio of the brief net of costs.

**Solution.**

The HRP core earns little on this data (annualized return of 1.1% with a volatility of 3.8%): HRP loads the
fixed-income blocks, the least volatile ones. The stock portfolio has an annualized return of 6.6% with a volatility of
18.5% and a maximum drawdown of −30.6%. The 60/40 ends at an annualized return of 3.7%, with the best Sharpe of the
three (0.49) and a maximum drawdown of −18.0%. That is the role of the core: lowering the portfolio's risk, not raising
its return. In the real version, each block is a UCITS ETF with its KID (7.4.3), and its TER and its dividend policy
go into the costs and into IRPF.

```python
u4 = pd.read_parquet("universo4_precios.parquet").set_index("fecha")
NUCLEO = ["RV_EEUU", "RV_EUR", "RV_JAP", "RV_EMG", "RF_GLB_C",
          "RF_T37", "INM_GLB"]
r4 = u4[NUCLEO].pct_change()
nuc = []
for a, b in zip(REB[:-1], REB[1:]):
    hist = r4.loc[:fechas[a]].iloc[-252:].to_numpy()
    w = hrp(np.cov(hist, rowvar=False))
    nuc.append(float(w @ (u4.loc[fechas[b], NUCLEO]
                          / u4.loc[fechas[a], NUCLEO] - 1)))
nuc = pd.Series(nuc, index=mercado.index)
sat = pruebas[pre]["neto"]
mix = {"ETF core": nuc, "satellite": sat,
       "60/40": 0.6 * nuc + 0.4 * sat}


def resumen(r):
    eq = (1 + r).cumprod()
    return {"tae_%": 100 * (eq.iloc[-1] ** (12 / len(r)) - 1),
            "vol_%": 100 * r.std() * np.sqrt(12),
            "sharpe": r.mean() / r.std() * np.sqrt(12),
            "max_caida_%": 100 * (eq / eq.cummax() - 1).min()}


pd.DataFrame({k: resumen(v) for k, v in mix.items()}).T.round(2)
```
```output
           tae_%  vol_%  sharpe  max_caida_%
ETF core    1.12   3.76    0.31       -14.07
satellite   6.57  18.46    0.44       -30.64
60/40       3.68   8.05    0.49       -18.02
```

3. In the defense, a committee member asks why you do not deliver the ridge with equal weights, which is the best of the
   six trials. What do you answer?

**Solution.**

That the brief set LightGBM and HRP before looking at results, and that the report presents the six trials, with the
ridge as the mandatory reference. That the ridge with equal weights is the best in this sample, but with a DSR of 0.727
and a PBO of 0.30 there is not enough evidence that it is outside it; changing configuration after seeing the result
and presenting it as if it were the chosen one is selecting after the fact, and that fails. The honest thing is what the
report does: say that, with this data, no combination demonstrates an edge after costs and taxes, propose the ridge as
a candidate for a new validation with later data and present none as tradable with money.

[^1]: SEC, “EDGAR Application Programming Interfaces (APIs)” (https://www.sec.gov/search-filings/edgar-application-programming-interfaces):
    “These APIs do not require any authentication or API keys to access”; description of `companyfacts` and `frames`.
    Accessed October 24, 2026.

[^2]: Massive, “Pricing” (https://massive.com/pricing): “Stocks Advanced”, 199 USD a month, “Individual use only”, with
    more than 20 years of history; “Financials & Ratios”, 29 USD a month, “Individual use only”. Accessed October 24,
    2026.

## Self-assessment

1. The skeleton's audit measures the monthly IC of each feature in 2012-2014 and sets aside `revision_analistas` (IC 0.096, t = 10.8). Why is it a leak and not a good feature?

   - A. Because its IC is negative
   - B. Because it is an analyst feature, and those never work
   - C. Because it has missing data
   - D. Because in a panel where honest features give ICs of 0.02 to 0.04, an IC of almost 0.10 with t > 10 is the signature of future information: the vendor dates it at the start of the week it covers, and it contains returns after the decision date

   *Answer D: Because in a panel where honest features give ICs of 0.02 to 0.04, an IC of almost 0.10 with t > 10 is the signature of future information: the vendor dates it at the start of the week it covers, and it contains returns after the decision date.* The panel's README describes the trap: the revision mixes in the returns of the first two sessions of the following week. A demonstrable look-ahead fails on its own; the audit uses only the first training window so that the filter does not look at the future either.

2. In the walk-forward, the LightGBM LambdaRank has an IC of 0.0096 (t = 1.29) and a rank correlation of 0.28 with `vol_63d`; the ridge, 0.0221 (t = 3.15). What does the project do?

   - A. It says so: on this data the linear reference wins, and the ranker has mostly learned to sort by volatility, as in 5.5.6; the model of the brief is presented with the comparison, without tuning hyperparameters until it wins
   - B. It searches the ranker's hyperparameters until it beats the ridge
   - C. It removes the ridge from the report, because the brief asks for LightGBM
   - D. It changes the label until the IC goes up

   *Answer A: It says so: on this data the linear reference wins, and the ranker has mostly learned to sort by volatility, as in 5.5.6; the model of the brief is presented with the comparison, without tuning hyperparameters until it wins.* The naive reference is part of the course method. Tuning the ranker against the same period until it wins adds trials to the log, and the DSR penalizes them. What you cannot do is hide the comparison.

3. With IRPF, the portfolio of the brief ends with an annualized return of 5.2%; with a two-month lockout on buying back what was sold at a loss, 4.6%. Why does the lockout come out worse?

   - A. Because the lockout pays more IRPF
   - B. Because the two-month rule defers the loss, it does not eliminate it; the lockout moves the portfolio away from the one the model asks for and that difference costs more than it saves in taxes paid early
   - C. Because the lockout breaks the law
   - D. Because with the lockout costs are not applied

   *Answer B: Because the two-month rule defers the loss, it does not eliminate it; the lockout moves the portfolio away from the one the model asks for and that difference costs more than it saves in taxes paid early.* The deferred loss is recognized when the repurchased shares are sold, so the saving from the lockout is only one of timing. In this portfolio, with high monthly turnover, the cost of not following the ranking is greater. It is a result of this data, not a general rule, and it is not advice.

4. The oracle (the panel's true alpha) with equal weights beats the universe by 2.0% a year; with HRP, by 0.0%. What does it teach?

   - A. That the oracle is computed wrongly
   - B. That HRP always destroys value
   - C. That HRP decides weights by risk, not by expected return: it loads the least volatile stocks and, measured against an equal-weighted universe in a rising market, it eats up the selection; if the goal is alpha, HRP is a risk control, not the source
   - D. That you have to rebalance every week

   *Answer C: That HRP decides weights by risk, not by expected return: it loads the least volatile stocks and, measured against an equal-weighted universe in a rising market, it eats up the selection; if the goal is alpha, HRP is a risk control, not the source.* The panel's known answer separates the two pieces: the selection adds value, the risk weighting changes the profile (similar Sharpe, less weight in volatile names). The report has to say which of the two is evaluated against which reference.

5. The portfolio rebalances at month-end. How is the paper trading set up?

   - A. Four weeks from any day: one rebalance is enough
   - B. The second rebalance is simulated with the backtest
   - C. It rebalances every week during paper
   - D. It starts a few days before a month-end and runs until after the next one: at least four weeks and two rebalances with real paper orders, daily reconciliation and the lot record squared with the broker

   *Answer D: It starts a few days before a month-end and runs until after the next one: at least four weeks and two rebalances with real paper orders, daily reconciliation and the lot record squared with the broker.* What is evaluated on paper is the full cycle: data, signal, selection, orders, executions, lots and report. With a single rebalance there is no sale of what was already held, which is where the two-month rule and lot errors show up.

## Key points

- C3 is a long portfolio of 40 stocks chosen every month by a ranking model, weighted with HRP and rebalanced with costs and IRPF; ETFs come in as a UCITS core, not as a satellite.
- The leakage audit comes before the model: on the level 5 panel it sets aside the analyst revision (IC 0.096, t = 10.8) looking only at the first training window.
- In the walk-forward, the ridge (IC 0.022) beats LightGBM as a regressor (0.012) and as LambdaRank (0.010), which also loads up on volatility. The project presents it with the comparison, it does not hide it.
- With six logged trials, the best (ridge with equal weights, IR 0.41) has a DSR of 0.727; the one of the brief, an IR of −0.01. There is no evidence of an edge after costs and selection.
- IRPF takes the portfolio's annualized return from 6.5% to 5.2%; almost half of the realized losses are deferred by the two-month rule, which moves taxable base between years without changing the total.
- A tax engine without the rule brings losses forward (€5,414 in 2015); blocking repurchases avoids part of the deferrals but, in this portfolio, costs more than it saves.

## Resources

### Research

- [Alpha Factors and Alpha Trading in Python](https://pythonparatrading.com/en/alpha-factors-and-alpha-trading-in-python/) · Research-0007 · 5 min

### Documentation

- [LightGBM 4.7.0: parameters (lambdarank, label_gain, lambdarank_truncation_level)](https://lightgbm.readthedocs.io/en/stable/Parameters.html) · lightgbm.readthedocs.io
- [Law 35/2006 on IRPF (Spain's personal income tax), consolidated text (BOE, in Spanish): arts. 33.5 f), 37.2, 49, 66 and 76](https://www.boe.es/buscar/act.php?id=BOE-A-2006-20764) · boe.es
- [DGT (Spain's Directorate General for Taxation), binding ruling V5316-26: how the two-month rule applies (in Spanish)](https://petete.tributos.hacienda.gob.es/consultas/?num_consulta=V5316-26) · petete.tributos.hacienda.gob.es
- [Kenneth R. French: Data Library (factors and benchmark portfolios)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) · mba.tuck.dartmouth.edu

### Books

- [Marcos López de Prado, Machine Learning for Asset Managers (Cambridge University Press, 2020)](https://www.cambridge.org/core/elements/machine-learning-for-asset-managers/6D9211305EA2E425D33A9F38D0AE3545) · cambridge.org

### Data

- [Level 5 data: weekly features of the stock panel (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet

Previous lesson: [7.6.3 · Capstone C2: systematic options book, volatility surface and real-time Greeks](https://pythonparatrading.com/en/python-trading-course/level-7/capstone-options-book-volatility-greeks/)

Next lesson: [7.6.5 · Capstone C4: a research platform with LLMs and alternative data](https://pythonparatrading.com/en/python-trading-course/level-7/capstone-llm-research-platform/)