Lesson 5.10.3 · 70 min

Ensembles, signal combination and synthetic data

Combining several signals is the cheapest and most reliable improvement in quantitative research, with one condition: their errors must not be the same. This lesson combines four signals from the level 5 panel with weights estimated only from the past, measures how much their errors diversify and why equal weights usually beat estimated ones. Then, the fashionable synthetic data: GANs and diffusion, what they reproduce, how they are evaluated and why they do not manufacture signal.

Fig. — Out-of-sample IC (2015-2025) of four level 5 panel signals and of three means of standardised signals

03/03

Module: 5.10 · Interpretability, robustness and model risk
Duration: 70 min
Prerequisites: 5.4.3 · Walk-forward with retraining: windows and frequency, 5.5.3 · Gradient boosting: XGBoost, LightGBM and CatBoost, 5.7.4 · Graph neural networks for asset relationships, 2.5.7 · Synthetic data for stress testing: GBM, GARCH and bootstrapping
Author: Jesús Cuesta
Updated: 04 Nov 2026

Level 5 lessons

Why combine

In 1969, Bates and Granger showed that the mean of two forecasts of airline passengers was better than either of them on its own. The idea is the same as that of a portfolio: two assets with the same expected return and imperfect correlation give, together, less variance. With signals, what gets diversified is the errors.

For standardized signals (mean 0 and standard deviation 1 across stocks each week), the IC of their mean is computed from two things: the IC of each one and the mean correlation between them. With $k$ signals with IC $\rho_i$ and correlations $c_{ij}$ ( $c_{ii} = 1$ ):

\text{IC}\Big(\tfrac{1}{k}\textstyle\sum_i s_i\Big) = \frac{\overline{\rho}}{\sqrt{\overline{c}}}, \qquad \overline{\rho} = \frac{1}{k}\sum_i \rho_i, \quad \overline{c} = \frac{1}{k^2}\sum_{i,j} c_{ij}

If all the signals are the same, $\overline{c} = 1$ and the mean gains nothing. If they are independent and have the same IC, $\overline{c} = 1/k$ and the IC is multiplied by $\sqrt{k}$ . Everything that happens in practice lies in between. The formula also says what adding a bad signal costs: it lowers $\overline{\rho}$ .

Mermaid source

flowchart TB
  D[Data and features] --> A[Ridge signal]
  D --> B[LightGBM signal]
  D --> C[Fixed factors]
  G[Customer graph] --> E[Graph signal]
  A --> M[Combine]
  B --> M
  C --> M
  E --> M
  M --> P[Portfolio]

The first three signals come from the same panel features; the fourth, from another source: the customer-supplier graph of lesson 5.7.4. That difference will decide almost everything.

Four out-of-sample signals

The signals are generated with the walk-forward of lesson 5.4.3: every January from 2015 to 2025 the models are retrained on all the weeks whose label ended before the decision date, and the year is predicted. The features are in cross-sectional normal rank; the label is the return of the following week minus its mean.

Ridge (α = 10) with the 21 features, the reference model of the README.
LightGBM with the same features (lesson 5.5.3), small and heavily regularized.
Fixed factors: the sum of six features with the sign given by the literature (momentum, value, SUE and sentiment with sign +; last week's return and volatility with sign −). It estimates nothing.
Graph: the weighted mean of the customers' return in the previous week, minus the mean of their sector (lesson 5.7.4). It is 0 for the 101 stocks without customers.

IN [01]python

import numpy as np
import pandas as pd
from scipy.stats import norm, rankdata
from sklearn.linear_model import Ridge
import lightgbm as lgb

feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet")
feat = feat.sort_values(["fecha", "ticker"])
etiq = etiq.sort_values(["fecha", "ticker"])
FEATS = [c for c in feat.columns if c not in
         ("fecha", "ticker", "sector", "revision_analistas")]
fechas = pd.DatetimeIndex(feat["fecha"].unique())
nW, nA, nF = len(fechas), feat["ticker"].nunique(), len(FEATS)

F = feat[FEATS].to_numpy(dtype=float).reshape(nW, nA, nF)
with np.errstate(invalid="ignore", divide="ignore"):
    rango = rankdata(F, axis=1, nan_policy="omit")
    n_ = np.isfinite(F).sum(axis=1, keepdims=True)
    G = np.nan_to_num(norm.ppf((rango - 0.5) / n_))
R = etiq["ret_sig_semana"].to_numpy().reshape(nW, nA)
fin = etiq["fin_sig_semana"].to_numpy().reshape(nW, nA)[:, 0]
con_feat = np.isfinite(F[:, :, 0]).all(axis=1)
con_etiq = np.isfinite(R).all(axis=1)
valida = con_feat & con_etiq
Yc = np.zeros_like(R)
Yc[valida] = R[valida] - R[valida].mean(axis=1, keepdims=True)
anyo = fechas.year.to_numpy()
pru = np.flatnonzero(con_feat & (anyo >= 2015))
evaluables = pru[con_etiq[pru]]

def ultima_conocida(w):
    """Last week whose label ended before date w."""
    return int(np.searchsorted(fin, fechas[w], side="left")) - 1

print(f"{nW} weeks x {nA} stocks x {nF} features | test: "
      f"{len(evaluables)} weeks from {fechas[pru[0]].date()}")

OUT [01]Panel as arrays and the time rule

730 weeks x 400 stocks x 21 features | test: 574 weeks from 2015-01-02

IN [02]python

senal = {k: np.full((nW, nA), np.nan) for k in ("ridge", "LightGBM")}
for a in range(2015, 2026):
    semanas = pru[anyo[pru] == a]
    ent = np.flatnonzero(valida[:ultima_conocida(semanas[0]) + 1])
    X_ent, y_ent = G[ent].reshape(-1, nF), Yc[ent].reshape(-1)
    ridge = Ridge(alpha=10.0).fit(X_ent, y_ent)
    gbm = lgb.LGBMRegressor(
        n_estimators=200, learning_rate=0.03, num_leaves=15,
        min_child_samples=500, subsample=0.5, subsample_freq=1,
        colsample_bytree=0.8, reg_lambda=10.0, random_state=0,
        n_jobs=4, deterministic=True, force_row_wise=True,
        verbose=-1).fit(X_ent, y_ent)
    for w in semanas:
        senal["ridge"][w] = ridge.predict(G[w])
        senal["LightGBM"][w] = gbm.predict(G[w])
print("training weeks in 2015 and in 2025:",
      len(np.flatnonzero(valida[:ultima_conocida(pru[0]) + 1])),
      len(ent))

OUT [02]Ridge and LightGBM in walk-forward, retrained every January

training weeks in 2015 and in 2025: 154 676

IN [03]python

j = {f: i for i, f in enumerate(FEATS)}
signos = {"mom_12_1": 1, "ret_1s": -1, "valor_bp": 1, "sue": 1,
          "sentimiento": 1, "vol_63d": -1}
senal["factors"] = sum(s * G[:, :, j[f]] for f, s in signos.items())

act = pd.read_csv("panel5_activos.csv").sort_values("ticker")
pos = {t: i for i, t in enumerate(act["ticker"])}
g = pd.read_csv("grafo5_relaciones.csv")
cli = g[g["tipo"] == "cliente"]
Wc = np.zeros((nA, nA))
Wc[cli["origen"].map(pos), cli["destino"].map(pos)] = cli["peso"]
sector = pd.factorize(act["sector"])[0]
Rs = R.copy()
for s_ in np.unique(sector):
    m = sector == s_
    with np.errstate(invalid="ignore"):
        Rs[:, m] = R[:, m] - R[:, m].mean(axis=1, keepdims=True)
X_prev = np.zeros_like(R)
X_prev[1:] = np.nan_to_num(Rs[:-1])    # the week ending at w
senal["graph"] = X_prev @ Wc.T
print({k: int(np.isfinite(v[pru]).all()) for k, v in senal.items()})

OUT [03]Fixed factors and the graph signal

{'ridge': 1, 'LightGBM': 1, 'factors': 1, 'graph': 1}

The four signals are converted to cross-sectional normal rank every week, so that the combination does not depend on their scale (the ridge's is in basis points; the factors', in units). Then, their weekly IC, their t and two measures of how similar they are: the mean correlation between their scores and the correlation between their IC series.

IN [04]python

def rango_normal(p):
    out = np.full_like(p, np.nan)
    for w in pru:
        out[w] = norm.ppf((rankdata(p[w]) - 0.5) / nA)
    return out

def ic_semanal(p, semanas=evaluables):
    return np.array([np.corrcoef(rankdata(p[w]), rankdata(R[w]))[0, 1]
                     for w in semanas])

def resumen(ic):
    et = ic.std(ddof=1) / np.sqrt(len(ic))
    return {"IC": round(float(ic.mean()), 4),
            "standard error": round(float(et), 4),
            "t": round(float(ic.mean() / et), 1)}

Z = {k: rango_normal(v) for k, v in senal.items()}
nombres = list(Z)
ics = {k: ic_semanal(Z[k]) for k in nombres}
print(pd.DataFrame({k: resumen(v) for k, v in ics.items()}).T)
C = np.mean([np.corrcoef(np.stack([Z[k][w] for k in nombres]))
             for w in evaluables], axis=0)
print("\nmean correlation between scores")
print(pd.DataFrame(C, nombres, nombres).round(2))
print("\ncorrelation between the weekly IC series")
print(pd.DataFrame(ics).corr().round(2))

OUT [04]IC of each signal and how similar they are

              IC  standard error    t
ridge     0.0185          0.0038  4.9
LightGBM  0.0145          0.0032  4.6
factors   0.0149          0.0039  3.8
graph     0.0090          0.0021  4.3

mean correlation between scores
          ridge  LightGBM  factors  graph
ridge      1.00      0.63     0.68   -0.0
LightGBM   0.63      1.00     0.44    0.0
factors    0.68      0.44     1.00    0.0
graph     -0.00      0.00     0.00    1.0

correlation between the weekly IC series
          ridge  LightGBM  factors  graph
ridge      1.00      0.84     0.46  -0.00
LightGBM   0.84      1.00     0.36  -0.02
factors    0.46      0.36     1.00   0.11
graph     -0.00     -0.02     0.11   1.00

The ridge gives the IC of the README (0.0185, t = 4.9). LightGBM and the fixed factors come in below it, and the graph, with half the IC, has a similar t because its weekly IC is more stable. What matters is in the correlations: the three signals that come from the features are very similar (scores between 0.44 and 0.68; IC series of ridge and LightGBM, 0.84), and the graph resembles none of them (0.00).

The mean and its predicted IC

IN [05]python

combos = {"ridge + LightGBM + factors": ["ridge", "LightGBM",
                                         "factors"],
          "ridge + graph": ["ridge", "graph"],
          "all four": nombres}
filas = {}
for nombre, ks in combos.items():
    idx = [nombres.index(k) for k in ks]
    predicho = (np.mean([ics[k].mean() for k in ks])
                / np.sqrt(C[np.ix_(idx, idx)].mean()))
    ic_m = ic_semanal(sum(Z[k] for k in ks) / len(ks))
    dif = ic_m - ics["ridge"]
    filas[nombre] = {**resumen(ic_m),
                     "predicted IC": round(predicho, 4),
                     "diff. vs ridge": round(dif.mean(), 4),
                     "t diff.": round(dif.mean() / dif.std(ddof=1)
                                      * np.sqrt(len(dif)), 1)}
pd.DataFrame(filas).T

OUT [05]Measured IC of several means against the formula

                                IC  standard error    t  predicted IC  diff. vs ridge  t diff.
ridge + LightGBM + factors  0.0186          0.0036  5.2        0.0188          0.0001      0.1
ridge + graph               0.0194          0.0030  6.4        0.0194          0.0010      0.5
all four                    0.0205          0.0034  6.0        0.0207          0.0020      1.1

The formula is right to within a few ten-thousandths in all three means. And it explains the results:

Three signals from the same features: high mean correlation, so the denominator barely drops below 1 and the mean of their ICs (below the ridge) stays almost the same. The mean does not improve on the ridge.
Ridge + graph: zero correlation. The mean of their ICs is low (0.0137), but it is divided by $\sqrt{0.5}$ and rises above the ridge. The most visible change is the t: from 4.9 to 6.4.
All four: 0.0205 with t = 6.0, the best IC in the table.

The t rises much more than the IC because diversification acts mainly on the week-by-week error: the ridge's bad weeks are not the graph's, and the weekly IC series of the mean is more stable. The paired difference in mean IC against the ridge, on the other hand, has t ≈ 1: the mean IC improves a little and the stability improves a lot. For a portfolio, the second is what counts most (lesson 5.1.3: the Sharpe of a strategy is, roughly, the IC times the square root of the breadth, divided by the variability of the IC).

With what weights

The simple mean gives the same weight to a signal with an IC of 0.018 as to one with 0.009. It looks improvable: weight by past IC or learn the weights with a regression (stacking, Wolpert, 1992; Breiman, 1996). The honesty condition is the usual one: the weights for week $w$ can only use weeks whose label has already ended, and with predictions that were out of sample at the time. A year of predictions is needed to start, so the comparison runs from 2016 to 2025.

Four schemes:

Scheme	Weights	Estimated
Mean	1/4 each	nothing
IC weights	proportional to each signal's past mean IC (0 if negative)	4 means
Nested stacking	regression of the label on the 4 signals, with past weeks	4 coefficients
Hindsight	the same regression with all the weeks of 2016-2025	4 coefficients, with the future

IN [06]python

ic_w = {k: np.full(nW, np.nan) for k in nombres}
for k in nombres:
    ic_w[k][evaluables] = ics[k]
por_ic = np.full((nW, nA), np.nan)
stack = np.full((nW, nA), np.nan)
pesos_stack = {}
for w in pru:
    pasado = evaluables[evaluables <= ultima_conocida(w)]
    if len(pasado) < 52:
        continue
    m = np.clip([np.mean(ic_w[k][pasado]) for k in nombres], 0, None)
    por_ic[w] = sum(m[i] / m.sum() * Z[k][w]
                    for i, k in enumerate(nombres))
    Xs = np.stack([Z[k][pasado].ravel() for k in nombres], axis=1)
    b = np.linalg.lstsq(Xs, Yc[pasado].ravel(), rcond=None)[0]
    stack[w] = sum(b[i] * Z[k][w] for i, k in enumerate(nombres))
    pesos_stack[fechas[w]] = b / b.sum()

ev2 = np.array([w for w in evaluables if np.isfinite(stack[w]).all()])
Xr = np.stack([Z[k][ev2].ravel() for k in nombres], axis=1)
b_retro = np.linalg.lstsq(Xr, Yc[ev2].ravel(), rcond=None)[0]
media4 = sum(Z[k] for k in nombres) / 4
esquemas = {"ridge": Z["ridge"], "mean": media4,
            "IC weights": por_ic, "nested stacking": stack,
            "hindsight": sum(b_retro[i] * Z[k]
                             for i, k in enumerate(nombres))}
ic2 = {k: ic_semanal(p, ev2) for k, p in esquemas.items()}
tabla = pd.DataFrame({k: resumen(v) for k, v in ic2.items()}).T
for k, v in ic2.items():
    dif = v - ic2["mean"]
    et = dif.std(ddof=1) / np.sqrt(len(dif))
    tabla.loc[k, "diff. vs mean"] = round(dif.mean(), 4)
    tabla.loc[k, "t diff."] = (round(dif.mean() / et, 1)
                               if et > 1e-6 else np.nan)
print(f"{len(ev2)} weeks from {fechas[ev2[0]].date()}")
print(tabla.to_string())
print("hindsight weights:",
      dict(zip(nombres, (b_retro / b_retro.sum()).round(2).tolist())))
ultimos = pesos_stack[fechas[ev2[-1]]].round(2).tolist()
print("stacking, last week:", dict(zip(nombres, ultimos)))

OUT [06]Equal weights, past-IC weights, stacking and hindsight

521 weeks from 2016-01-08
                     IC  standard error    t  diff. vs mean  t diff.
ridge            0.0175          0.0040  4.4        -0.0021     -1.1
mean             0.0196          0.0036  5.5         0.0000      NaN
IC weights       0.0184          0.0037  4.9        -0.0012     -2.4
nested stacking  0.0178          0.0037  4.8        -0.0018     -1.3
hindsight        0.0201          0.0036  5.6         0.0005      0.4
hindsight weights: {'ridge': 0.62, 'LightGBM': -0.02, 'factors': 0.04, 'graph': 0.36}
stacking, last week: {'ridge': 0.66, 'LightGBM': 0.02, 'factors': 0.02, 'graph': 0.31}

Fig. — Nested stacking weights (regression of the label on the four signals using past weeks), at the end of each quarter from 2016 to 2025

The simple mean beats the two schemes that estimate weights from the past, and ties with the weights chosen by looking at the future, which are the best possible linear combination in this sample. It is the forecast combination puzzle: in studies with macroeconomic data, the simple mean beats “optimal” combinations again and again. Smith and Wallis (2009) give the explanation: the optimal sample weights carry estimation error, and when the signals have small, similar ICs, that error costs more than what is gained by choosing well. The figure shows the stacking weights over time: they move with the noise of each year, and in some stretches they give negative weight to signals that are good out of sample.

From signal to portfolio

The IC is the metric of the signal; the portfolio also pays for turnover. A long-short decile portfolio, equally weighted and rebalanced every week, with 10 bp per side on turnover (as in the README):

IN [07]python

def larga_corta(p, semanas=ev2, coste=0.001):
    rend, rot, previo = [], [], np.zeros(nA)
    for w in semanas:
        rk = rankdata(p[w])
        q = np.where(rk > 0.9 * nA, 1.0, 0.0)
        q = q / q.sum() - np.where(rk <= 0.1 * nA, 1.0, 0.0) \
            / (rk <= 0.1 * nA).sum()
        rend.append(q @ R[w])
        rot.append(np.abs(q - previo).sum())
        previo = q
    rend, rot = np.array(rend), np.array(rot)
    neto = rend - coste * rot
    return {"annual ret. %": 5200 * rend.mean(),
            "gross Sharpe": np.sqrt(52) * rend.mean() / rend.std(),
            "weekly turnover %": 100 * rot[1:].mean(),
            "net Sharpe": np.sqrt(52) * neto.mean() / neto.std()}

pd.DataFrame({k: larga_corta(Z[k] if k in Z else media4)
              for k in ("ridge", "graph", "mean of the four")}
             ).T.round(2)

OUT [07]Long-short by deciles: ridge, graph and the mean of the four

                  annual ret. %  gross Sharpe  weekly turnover %  net Sharpe
ridge                     10.77          1.13             262.72       -0.30
graph                      6.06          1.06             322.34       -1.87
mean of the four          10.04          1.18             272.89       -0.49

Combining does not fix costs. The mean of the four signals has a gross Sharpe similar to the ridge's (1.18 against 1.13) and turns over somewhat more (273% against 263% a week), so net of costs it still does not make money (−0.49 against −0.30). The graph signal alone, with a gross Sharpe of 1.06, is the one that comes out worst net (−1.87): it changes sign every week with the customers' return and turns over 322%. In all three cases, the turnover of a weekly cross-sectional signal eats up an edge of this size (lesson 5.1.3). Combining improves the ingredient; the portfolio recipe (rebalancing frequency, no-trade bands, cost per trade) is another lesson.

Synthetic data: GANs and diffusion

The other half of the section is generating data. Lesson 2.5.7 used explicit-model generators (GBM, GARCH, bootstrap) for stress tests, and all the data in this course come from models with published parameters. Deep generators promise something else: learning the distribution of the series directly from the data, without writing down the model.

GANs (Goodfellow et al., 2014): a generator turns noise into series and a discriminator tries to tell them apart from the real ones; they are trained against each other. In finance, Quant GANs (Wiese et al., 2020) uses temporal convolutional networks in both and claims to reproduce volatility clustering, the leverage effect and autocorrelations; TimeGAN (Yoon et al., 2019) adds a supervised loss on the temporal dynamics.
Diffusion (Ho et al., 2020): the model learns to remove noise from data to which noise has been added step by step, and generates from pure noise. Takahashi and Mizuno (2024) apply diffusion models to series of prices, volumes and bid-ask spreads converted into images with a wavelet transform and claim to reproduce fat tails, volatility clustering and seasonality; CoFinDiff (Tanaka et al., 2025) conditions the generation on a trend and a realized volatility.

What these papers claim is checked with three families of tests, which are also the ones highlighted by the review of Assefa et al. (2020) on synthetic data in finance:

Test	Question	How
Stylized facts	Does it reproduce tails, volatility clustering, near-zero autocorrelation of returns?	kurtosis, extreme quantiles, autocorrelation of r and of \|r\|
Memorization	Does it copy the training sample?	distance from each generated series to the nearest training one, against that of new real data
External discriminator	Can it be told apart from the real data?	a classifier outside training; AUC 0.5 = indistinguishable

A small GAN against the bootstrap

To get hands-on, a minimal GAN in PyTorch that generates windows of 20 sessions of returns of the indice asset in vol5_diario.parquet (the level's volatility lab: GARCH-HAR with t tails and leverage effect). It is trained on 2008-2019 (3,002 overlapping, standardized windows) and compared with the 2020-2025 data, with normal noise and with the stationary bootstrap of lesson 2.5.7 (blocks of 10 sessions on average). A real generator would train much longer and with temporal networks; this one takes a few seconds on CPU.

IN [08]python

import torch
from scipy.stats import kurtosis

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
vol = pd.read_parquet("vol5_diario.parquet")
r = vol[vol["activo"] == "indice"].set_index("fecha")["rendimiento"]
L, sd = 20, r[:"2019-12-31"].std()
ventanas = lambda x: np.lib.stride_tricks.sliding_window_view(x, L)
V_ent = ventanas(r[:"2019-12-31"].to_numpy() / sd)
V_pru = ventanas(r["2020-01-01":].to_numpy() / sd)

def red(entrada, salida):
    return torch.nn.Sequential(
        torch.nn.Linear(entrada, 64), torch.nn.LeakyReLU(0.2),
        torch.nn.Linear(64, 64), torch.nn.LeakyReLU(0.2),
        torch.nn.Linear(64, salida))

gen, disc = red(16, L), red(L, 1)
o_g = torch.optim.Adam(gen.parameters(), lr=2e-4, betas=(0.5, 0.999))
o_d = torch.optim.Adam(disc.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = torch.nn.BCEWithLogitsLoss()
reales = torch.tensor(V_ent, dtype=torch.float32)
azar = torch.Generator().manual_seed(0)
uno, cero = torch.ones(128, 1), torch.zeros(128, 1)
for paso in range(3000):
    lote = reales[torch.randint(0, len(reales), (128,), generator=azar)]
    falsas = gen(torch.randn(128, 16, generator=azar))
    o_d.zero_grad()
    p_d = bce(disc(lote), uno) + bce(disc(falsas.detach()), cero)
    p_d.backward()
    o_d.step()
    o_g.zero_grad()
    p_g = bce(disc(falsas), uno)
    p_g.backward()
    o_g.step()
with torch.no_grad():
    V_gan = gen(torch.randn(3000, 16, generator=azar)).numpy()
V_gan = V_gan.astype(float)
print(f"windows: training {V_ent.shape}, test {V_pru.shape}, "
      f"GAN {V_gan.shape}")
print(f"final losses: discriminator {p_d.item():.2f}, "
      f"generator {p_g.item():.2f}")

OUT [08]A GAN of two networks with 64 neurons

windows: training (3002, 20), test (1489, 20), GAN (3000, 20)
final losses: discriminator 1.20, generator 0.80

The final losses do not say whether the GAN is good: in a GAN, the two networks can be in equilibrium and the generator can produce series that look nothing like the real ones in what matters. It has to be measured.

IN [09]python

rng = np.random.default_rng(0)

def bootstrap_estacionario(x, n, media=10):
    out = np.empty((n, L))
    for i in range(n):
        j, serie = rng.integers(len(x)), []
        while len(serie) < L:
            serie.append(x[j])
            j = (rng.integers(len(x)) if rng.random() < 1 / media
                 else (j + 1) % len(x))
        out[i] = serie
    return out

V_boot = bootstrap_estacionario(r[:"2019-12-31"].to_numpy() / sd, 3000)
V_norm = rng.standard_normal((3000, L))

def hechos(V):
    def ac(Y, k):
        return np.corrcoef(Y[:, :-k].ravel(), Y[:, k:].ravel())[0, 1]
    return {"kurtosis": kurtosis(V.ravel()),
            "0.5% quantile": np.quantile(V.ravel(), 0.005),
            "AC(1) of r": ac(V, 1),
            "AC(1) of |r|": ac(np.abs(V), 1),
            "AC(5) of |r|": ac(np.abs(V), 5)}

conjuntos = {"real 2008-2019": V_ent, "real 2020-2025": V_pru,
             "GAN": V_gan, "normal": V_norm, "bootstrap": V_boot}
pd.DataFrame({k: hechos(v) for k, v in conjuntos.items()}).T.round(3)

OUT [09]Stylized facts: real, GAN, normal and bootstrap

                kurtosis  0.5% quantile  AC(1) of r  AC(1) of |r|  AC(5) of |r|
real 2008-2019     3.793         -3.357      -0.026         0.088         0.028
real 2020-2025     1.865         -3.449       0.055         0.108         0.068
GAN                0.616         -2.709      -0.240         0.092         0.125
normal             0.023         -2.575      -0.002         0.005        -0.010
bootstrap          3.654         -3.357      -0.024         0.080         0.009

Fig. — Two stylised facts of 20-session windows of the indice asset: real, generated by a small GAN, normal and by stationary bootstrap

The GAN learned the scale of the returns and the lag-1 autocorrelation of |r| (0.092 against 0.088), but it fails at the rest: its kurtosis is a sixth of the real one (0.62 against 3.79: almost normal tails, with a 0.5% quantile of −2.71 against −3.36), it exaggerates the memory of |r| at five sessions (0.125 against 0.028) and it invents a lag-1 autocorrelation of returns of −0.24 that the real series does not have (−0.03). The bootstrap, which only reshuffles blocks of the history, reproduces the tails and the autocorrelation of returns by construction and keeps part of the clustering.

IN [10]python

from sklearn.neighbors import NearestNeighbors
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

vecino = NearestNeighbors(n_neighbors=1).fit(V_ent)

def resumen_ventana(V):
    a = np.abs(V)
    ac = [np.corrcoef(x[:-1], x[1:])[0, 1] for x in a]
    return np.c_[V.std(1), kurtosis(V, axis=1), ac, a.max(1), V.sum(1)]

X_real = resumen_ventana(V_pru[::5])          # 298 real windows
filas = {}
for k in ("real 2020-2025", "GAN", "normal", "bootstrap"):
    V = conjuntos[k]
    fila = {"nearest-neighbor distance (median)":
            np.median(vecino.kneighbors(V[:1000])[0])}
    if k != "real 2020-2025":
        X_s = resumen_ventana(V[:len(X_real)])
        X_c = np.r_[X_real, X_s]
        y_c = np.r_[np.zeros(len(X_real)), np.ones(len(X_s))]
        fila["discriminator AUC"] = cross_val_score(
            LogisticRegression(max_iter=1000), X_c, y_c, cv=5,
            scoring="roc_auc").mean()
    filas[k] = fila
pd.DataFrame(filas).T.round(3)

OUT [10]Memorization and external discriminator

                nearest-neighbor distance (median)  discriminator AUC
real 2020-2025                               2.963                NaN
GAN                                          2.969              0.754
normal                                       3.288              0.627
bootstrap                                    2.671              0.457

The distance from the GAN's windows to the nearest training window is that of new real data: it does not copy the sample (memorization is the risk of large generators with little data, and also a privacy problem when the data belong to clients). The external discriminator, a logistic regression with five summaries of each window, tells the GAN's windows apart from the real ones with an AUC of 0.75, better than the normal ones (0.63); the bootstrap's (0.46) it cannot tell apart. A deep generator that does not beat a bootstrap on these tests adds nothing to a stress test.

What synthetic data cannot do

Use	Does it make sense?	Why
Stress tests and scenarios	yes, with evaluated generators	more paths than history, with controlled tails and clustering (lesson 2.5.7)
Testing methods with a known answer	yes	it is what this course does: explicit models with planted effects
Sharing data without exposing clients	sometimes	if it is shown that there is no memorization
Augmenting data to train forecasting models	with great care	the generator comes from the same history: it adds no information about the future; any improvement is regularization and has to be compared with regularizing directly
Source of signal	no	a generator reproduces the distribution it saw, noise included; the edge a model “discovers” in synthetic data is the generator's

Deep generators are one more modeling tool, with the same underlying problem as any market model: a market is a single history that does not repeat, and what is not in it (the next crisis, with its new shape) is not in a generator trained on it either.

Exercises

With two uncorrelated signals, the optimal combination weights each one in proportion to its IC and reaches an IC of $\sqrt{\rho_1^2 + \rho_2^2}$ . Compute that ceiling for the ridge and the graph and compare it with the simple mean of the two and with stacking of just those two signals.

Solution

IN [11]python

r1, r2 = ics["ridge"].mean(), ics["graph"].mean()
techo = np.sqrt(r1 ** 2 + r2 ** 2)
optima = ic_semanal(r1 * Z["ridge"] + r2 * Z["graph"])
media2 = ic_semanal((Z["ridge"] + Z["graph"]) / 2)
st2 = np.full((nW, nA), np.nan)
for w in ev2:
    pasado = evaluables[evaluables <= ultima_conocida(w)]
    Xs = np.stack([Z[k][pasado].ravel() for k in ("ridge", "graph")], 1)
    b = np.linalg.lstsq(Xs, Yc[pasado].ravel(), rcond=None)[0]
    st2[w] = b[0] * Z["ridge"][w] + b[1] * Z["graph"][w]
print(f"theoretical ceiling {techo:.4f} | IC weights (known at the end) "
      f"{optima.mean():.4f} | mean {media2.mean():.4f}")
m2 = ic_semanal((Z["ridge"] + Z["graph"]) / 2, ev2).mean()
print(f"in 2016-2025: mean {m2:.4f} | stacking "
      f"{ic_semanal(st2, ev2).mean():.4f}")

OUT [11]Theoretical ceiling of ridge + graph

theoretical ceiling 0.0205 | IC weights (known at the end) 0.0205 | mean 0.0194
in 2016-2025: mean 0.0193 | stacking 0.0193

The weights proportional to the IC, computed over the whole period (in hindsight), reach the ceiling (0.0205); the simple mean stays at 0.0194, because it gives too much weight to the graph. But when the weights are truly estimated from the past (nested stacking, 2016-2025), they tie with the mean: 0.0193 both. The theoretical advantage of weighting well is eaten up by the error of estimating the weights, even with one signal clearly weaker than the other. The practical rule: equal weights unless there are strong reasons (and a lot of history) for something else.

Train the GAN for twice as many steps (6,000) with the same seed. Do the kurtosis, the lag-1 autocorrelation and the discriminator AUC improve?

Solution

IN [12]python

torch.manual_seed(0)
gen, disc = red(16, L), red(L, 1)
o_g = torch.optim.Adam(gen.parameters(), lr=2e-4, betas=(0.5, 0.999))
o_d = torch.optim.Adam(disc.parameters(), lr=2e-4, betas=(0.5, 0.999))
azar = torch.Generator().manual_seed(0)
for paso in range(6000):
    lote = reales[torch.randint(0, len(reales), (128,), generator=azar)]
    falsas = gen(torch.randn(128, 16, generator=azar))
    o_d.zero_grad()
    (bce(disc(lote), uno) + bce(disc(falsas.detach()), cero)).backward()
    o_d.step()
    o_g.zero_grad()
    bce(disc(falsas), uno).backward()
    o_g.step()
with torch.no_grad():
    V_gan2 = gen(torch.randn(3000, 16, generator=azar)).numpy()
V_gan2 = V_gan2.astype(float)
X_s = resumen_ventana(V_gan2[:len(X_real)])
auc = cross_val_score(LogisticRegression(max_iter=1000),
                      np.r_[X_real, X_s],
                      np.r_[np.zeros(len(X_real)), np.ones(len(X_s))],
                      cv=5, scoring="roc_auc").mean()
print({k: round(float(v), 3) for k, v in hechos(V_gan2).items()},
      "| AUC", round(float(auc), 3))

OUT [12]The same GAN with 6,000 steps

{'kurtosis': 1.076, '0.5% quantile': -2.428, 'AC(1) of r': -0.15, 'AC(1) of |r|': 0.088, 'AC(5) of |r|': 0.044} | AUC 0.773

With 6,000 steps the kurtosis rises somewhat (1.08) and the spurious autocorrelation falls (−0.15), but the discriminator tells it apart better than before (AUC 0.77 against 0.75). Training longer moves the figures, but neither monotonically nor toward what is sought: a GAN does not minimize a loss that measures the stylized facts, and its training oscillates. That is why serious work chooses the architecture (temporal networks that can represent volatility memory, tails in the input noise) and evaluates with tests fixed in advance, not with the discriminator loss.

A colleague proposes training the panel's signal model with ten times more weeks “generated with diffusion from the real panel”. What validation would you ask for before accepting it?

Solution

First, that the generator is fitted only on the training data of each walk-forward step (if it sees the test period, it is a leak through the back door). Second, a paired out-of-sample comparison between the model trained on real data and the one trained on real plus synthetic data, with the same walk-forward and their IC difference with its standard error. Third, a comparison with regularizing directly (more penalty, fewer trees, seed ensembles): if the synthetic data help, they most likely do so as regularization, and then the cheap option wins. And fourth, the generator's memorization tests: ten times more weeks that are near copies of the real ones only multiply the overfitting.

Each answer is folded under its question

Self-assessment

Ridge, LightGBM and a fixed factor combination, all with the same features, have ICs of 0.0185, 0.0145 and 0.0149, with correlations between their scores of 0.44 to 0.68. Their mean has an IC of 0.0186. Why does it not improve on the ridge?
- ABecause a mean of signals never improves on the best one
- BBecause LightGBM is badly trained
- CBecause the three models extract the same information from the same features: their errors are highly correlated and the mean dilutes the best one while diversifying almost nothing
- DBecause the mean should be multiplied by three
Show answer
Answer C. Because the three models extract the same information from the same features: their errors are highly correlated and the mean dilutes the best one while diversifying almost nothing
The IC of a mean of standardized signals is the mean of the ICs divided by the square root of the mean correlation between them. With high correlations the denominator barely drops below 1 and offsets just what the mean of the ICs (0.0160) loses against the best one. The formula predicts 0.0188 and 0.0186 is measured.
The customer-supplier graph signal has an IC of only 0.009, but zero correlation with the ridge. What happens when it is averaged with it?
- AThe IC of the mean rises and, above all, becomes more stable: the week-by-week errors of the two signals do not coincide and partly offset each other
- BThe IC falls because the graph signal is weaker
- CNothing changes: only the best signal matters
- DThe IC rises but its t falls
Show answer
Answer A. The IC of the mean rises and, above all, becomes more stable: the week-by-week errors of the two signals do not coincide and partly offset each other
With zero correlation, the IC of the mean is (0.0185 + 0.0090)/2/√0.5 ≈ 0.0194, more than the ridge alone. And the weekly IC series of the mean is less volatile, so its t rises much more than its mean: that is error diversification.
Nested stacking (a regression of the label on the signals, estimated only with past weeks) gives an IC of 0.0178 and the simple mean of the four signals, 0.0196. What is this?
- AA bug in the stacking
- BThe forecast combination puzzle: with little signal, the error of estimating the weights costs more than what is gained by choosing them, and equal weights usually win
- CProof that stacking is always worse
- DThat the simple mean uses future information
Show answer
Answer B. The forecast combination puzzle: with little signal, the error of estimating the weights costs more than what is gained by choosing them, and equal weights usually win
Smith and Wallis (2009) explain it this way: the optimal sample weights are estimated with a lot of noise when the signals have similar, small ICs. Equal weights are not estimated and do not carry that error.
A GAN trained on 20-day windows of returns has a kurtosis of 0.6 (the real series, 3.8), a lag-1 autocorrelation of −0.24 (the real one, −0.03) and an external discriminator tells it apart from the real data with an AUC of 0.75. The stationary bootstrap gives 0.46. What is the conclusion?
- AThat the GAN is better because it generates new data
- BThat AUC is not a good measure
- CThat the GAN should be used to train the models
- DThat this GAN does not reproduce the tails and invents a dependence that does not exist; for stress tests, the bootstrap of lesson 2.5.7 is better and much cheaper
Show answer
Answer D. That this GAN does not reproduce the tails and invents a dependence that does not exist; for stress tests, the bootstrap of lesson 2.5.7 is better and much cheaper
A generator is evaluated by what it reproduces and by what it invents. This small GAN fails at both; those in the literature (Quant GANs, diffusion) claim to do better, but they are evaluated with the same tests.
A vendor sells synthetic data generated with diffusion from the history of a market and claims that training on it improves forecasting models. What is the main objection?
- AThat diffusion is slower than a GAN
- BThat a generator trained on a single history contains no more information about the future than that history: it can help regularize or run stress tests, but it does not create signal, and it can copy the noise of the sample
- CThat synthetic data is always illegal
- DThat it needs a GPU
Show answer
Answer B. That a generator trained on a single history contains no more information about the future than that history: it can help regularize or run stress tests, but it does not create signal, and it can copy the noise of the sample
Augmenting data with a generator fitted to the same sample reuses the same information. If the model improves in an honest validation, the improvement comes from regularization, and it has to be compared with regularizing directly.

Key points

The IC of a mean of k standardized signals is the mean of their ICs divided by the square root of their mean correlation. It works on the data: it predicts 0.0207 for the mean of the four signals and 0.0205 is measured.
Signals extracted from the same features (ridge, LightGBM, fixed factors) correlate between 0.44 and 0.68 and their mean does not improve on the ridge. A signal from another source (the graph, correlation 0) does improve the mean and, above all, its stability.
Equal weights beat estimated ones (past IC, nested stacking) and tie with weights chosen in hindsight: the forecast combination puzzle. Hindsight weights are not a result, they are a leak.
Combining stabilizes the IC more than it raises it: the IC t of the mean of the four signals is 5.5 against 4.4 for the ridge alone, with a paired difference in mean IC that is not conclusive.
GANs and diffusion promise series with tails, volatility clustering and leverage. They are evaluated with stylized facts, memorization and an external discriminator. A small GAN fails where a bootstrap succeeds.
A generator fitted to one history does not create information about the future: it serves for stress tests, privacy and regularization, not as a source of signal.

Resources

Books

Data

Level 5 data: weekly panel features (Parquet)Download · panel5_features.parquet

Back to level 5

Why combine

Four out-of-sample signals

The mean and its predicted IC

With what weights

From signal to portfolio

Synthetic data: GANs and diffusion

A small GAN against the bootstrap

What synthetic data cannot do

Exercises

Self-assessment.

Key points.

Resources.

Books

Data

Self-assessment

Key points

Resources