---
title: "5.10.3 Ensembles, signal combination and synthetic data"
description: "Combining signals with out-of-sample weights, the diversification of their errors and a small GAN against the bootstrap: what synthetic data promises."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/ensembles-signal-combination-synthetic-data/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/ensembles-signal-combination-synthetic-data.png"
---
# 5.10.3 · Ensembles, signal combination and synthetic data
> Combining several signals is the cheapest and most reliable improvement in quantitative research, with one condition: their errors must not be the same. This lesson combines four signals from the level 5 panel with weights estimated only from the past, measures how much their errors diversify and why equal weights usually beat estimated ones. Then, the fashionable synthetic data: GANs and diffusion, what they reproduce, how they are evaluated and why they do not manufacture signal.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.10 · Interpretability, robustness and model risk
- Duration: 70 min
- Prerequisites: [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/), [5.5.3 · Gradient boosting: XGBoost, LightGBM and CatBoost](https://pythonparatrading.com/en/python-trading-course/level-5/xgboost-lightgbm-catboost-gradient-boosting/), [5.7.4 · Graph neural networks for asset relationships](https://pythonparatrading.com/en/python-trading-course/level-5/graph-neural-networks-for-assets/), [2.5.7 · Synthetic data for stress testing: GBM, GARCH and bootstrapping](https://pythonparatrading.com/en/python-trading-course/level-2/synthetic-data-for-stress-testing/)
## By the end of this lesson
- Generate out-of-sample signals in walk-forward (ridge, LightGBM, a fixed factor combination and the graph signal) and measure their IC and the correlation of their errors.
- Predict the IC of a mean of signals from their ICs and their correlations, and check it against the data.
- Combine with equal weights, past-IC weights and nested stacking, against weights chosen in hindsight, and explain the “forecast combination puzzle”.
- Train a small GAN on returns and evaluate it the way any generator is evaluated: stylized facts, memorization and an external discriminator, against the bootstrap of lesson 2.5.7.
## Why combine
In 1969, Bates and Granger showed that the mean of two forecasts of airline passengers was better than either of them
on its own. The idea is the same as that of a portfolio: two assets with the same expected return and imperfect
correlation give, together, less variance. With signals, what gets diversified is the **errors**.
For standardized signals (mean 0 and standard deviation 1 across stocks each week), the IC of their mean is computed
from two things: the IC of each one and the mean correlation between them. With $k$ signals with IC $\rho_i$ and
correlations $c_{ij}$ ($c_{ii} = 1$):
$$
\text{IC}\Big(\tfrac{1}{k}\textstyle\sum_i s_i\Big)
= \frac{\overline{\rho}}{\sqrt{\overline{c}}}, \qquad
\overline{\rho} = \frac{1}{k}\sum_i \rho_i, \quad
\overline{c} = \frac{1}{k^2}\sum_{i,j} c_{ij}
$$
If all the signals are the same, $\overline{c} = 1$ and the mean gains nothing. If they are independent and have the
same IC, $\overline{c} = 1/k$ and the IC is multiplied by $\sqrt{k}$. Everything that happens in practice lies in
between. The formula also says what adding a bad signal costs: it lowers $\overline{\rho}$.
```mermaid
flowchart TB
D[Data and features] --> A[Ridge signal]
D --> B[LightGBM signal]
D --> C[Fixed factors]
G[Customer graph] --> E[Graph signal]
A --> M[Combine]
B --> M
C --> M
E --> M
M --> P[Portfolio]
```
The first three signals come from the same panel features; the fourth, from another source: the customer-supplier
graph of lesson 5.7.4. That difference will decide almost everything.
## Four out-of-sample signals
The signals are generated with the walk-forward of lesson 5.4.3: every January from 2015 to 2025 the models are
retrained on all the weeks whose label ended before the decision date, and the year is predicted. The features are
in cross-sectional normal rank; the label is the return of the following week minus its mean.
- **Ridge** (α = 10) with the 21 features, the reference model of the README.
- **LightGBM** with the same features (lesson 5.5.3), small and heavily regularized.
- **Fixed factors**: the sum of six features with the sign given by the literature (momentum, value, SUE and sentiment
with sign +; last week's return and volatility with sign −). It estimates nothing.
- **Graph**: the weighted mean of the customers' return in the previous week, minus the mean of their sector
(lesson 5.7.4). It is 0 for the 101 stocks without customers.
```python
import numpy as np
import pandas as pd
from scipy.stats import norm, rankdata
from sklearn.linear_model import Ridge
import lightgbm as lgb
feat = pd.read_parquet("panel5_features.parquet")
etiq = pd.read_parquet("panel5_etiquetas.parquet")
feat = feat.sort_values(["fecha", "ticker"])
etiq = etiq.sort_values(["fecha", "ticker"])
FEATS = [c for c in feat.columns if c not in
("fecha", "ticker", "sector", "revision_analistas")]
fechas = pd.DatetimeIndex(feat["fecha"].unique())
nW, nA, nF = len(fechas), feat["ticker"].nunique(), len(FEATS)
F = feat[FEATS].to_numpy(dtype=float).reshape(nW, nA, nF)
with np.errstate(invalid="ignore", divide="ignore"):
rango = rankdata(F, axis=1, nan_policy="omit")
n_ = np.isfinite(F).sum(axis=1, keepdims=True)
G = np.nan_to_num(norm.ppf((rango - 0.5) / n_))
R = etiq["ret_sig_semana"].to_numpy().reshape(nW, nA)
fin = etiq["fin_sig_semana"].to_numpy().reshape(nW, nA)[:, 0]
con_feat = np.isfinite(F[:, :, 0]).all(axis=1)
con_etiq = np.isfinite(R).all(axis=1)
valida = con_feat & con_etiq
Yc = np.zeros_like(R)
Yc[valida] = R[valida] - R[valida].mean(axis=1, keepdims=True)
anyo = fechas.year.to_numpy()
pru = np.flatnonzero(con_feat & (anyo >= 2015))
evaluables = pru[con_etiq[pru]]
def ultima_conocida(w):
"""Last week whose label ended before date w."""
return int(np.searchsorted(fin, fechas[w], side="left")) - 1
print(f"{nW} weeks x {nA} stocks x {nF} features | test: "
f"{len(evaluables)} weeks from {fechas[pru[0]].date()}")
```
```output
730 weeks x 400 stocks x 21 features | test: 574 weeks from 2015-01-02
```
```python
senal = {k: np.full((nW, nA), np.nan) for k in ("ridge", "LightGBM")}
for a in range(2015, 2026):
semanas = pru[anyo[pru] == a]
ent = np.flatnonzero(valida[:ultima_conocida(semanas[0]) + 1])
X_ent, y_ent = G[ent].reshape(-1, nF), Yc[ent].reshape(-1)
ridge = Ridge(alpha=10.0).fit(X_ent, y_ent)
gbm = lgb.LGBMRegressor(
n_estimators=200, learning_rate=0.03, num_leaves=15,
min_child_samples=500, subsample=0.5, subsample_freq=1,
colsample_bytree=0.8, reg_lambda=10.0, random_state=0,
n_jobs=4, deterministic=True, force_row_wise=True,
verbose=-1).fit(X_ent, y_ent)
for w in semanas:
senal["ridge"][w] = ridge.predict(G[w])
senal["LightGBM"][w] = gbm.predict(G[w])
print("training weeks in 2015 and in 2025:",
len(np.flatnonzero(valida[:ultima_conocida(pru[0]) + 1])),
len(ent))
```
```output
training weeks in 2015 and in 2025: 154 676
```
```python
j = {f: i for i, f in enumerate(FEATS)}
signos = {"mom_12_1": 1, "ret_1s": -1, "valor_bp": 1, "sue": 1,
"sentimiento": 1, "vol_63d": -1}
senal["factors"] = sum(s * G[:, :, j[f]] for f, s in signos.items())
act = pd.read_csv("panel5_activos.csv").sort_values("ticker")
pos = {t: i for i, t in enumerate(act["ticker"])}
g = pd.read_csv("grafo5_relaciones.csv")
cli = g[g["tipo"] == "cliente"]
Wc = np.zeros((nA, nA))
Wc[cli["origen"].map(pos), cli["destino"].map(pos)] = cli["peso"]
sector = pd.factorize(act["sector"])[0]
Rs = R.copy()
for s_ in np.unique(sector):
m = sector == s_
with np.errstate(invalid="ignore"):
Rs[:, m] = R[:, m] - R[:, m].mean(axis=1, keepdims=True)
X_prev = np.zeros_like(R)
X_prev[1:] = np.nan_to_num(Rs[:-1]) # the week ending at w
senal["graph"] = X_prev @ Wc.T
print({k: int(np.isfinite(v[pru]).all()) for k, v in senal.items()})
```
```output
{'ridge': 1, 'LightGBM': 1, 'factors': 1, 'graph': 1}
```
The four signals are converted to cross-sectional normal rank every week, so that the combination does not depend on
their scale (the ridge's is in basis points; the factors', in units). Then, their weekly IC, their t and two measures
of how similar they are: the mean correlation between their scores and the correlation between their IC series.
```python
def rango_normal(p):
out = np.full_like(p, np.nan)
for w in pru:
out[w] = norm.ppf((rankdata(p[w]) - 0.5) / nA)
return out
def ic_semanal(p, semanas=evaluables):
return np.array([np.corrcoef(rankdata(p[w]), rankdata(R[w]))[0, 1]
for w in semanas])
def resumen(ic):
et = ic.std(ddof=1) / np.sqrt(len(ic))
return {"IC": round(float(ic.mean()), 4),
"standard error": round(float(et), 4),
"t": round(float(ic.mean() / et), 1)}
Z = {k: rango_normal(v) for k, v in senal.items()}
nombres = list(Z)
ics = {k: ic_semanal(Z[k]) for k in nombres}
print(pd.DataFrame({k: resumen(v) for k, v in ics.items()}).T)
C = np.mean([np.corrcoef(np.stack([Z[k][w] for k in nombres]))
for w in evaluables], axis=0)
print("\nmean correlation between scores")
print(pd.DataFrame(C, nombres, nombres).round(2))
print("\ncorrelation between the weekly IC series")
print(pd.DataFrame(ics).corr().round(2))
```
```output
IC standard error t
ridge 0.0185 0.0038 4.9
LightGBM 0.0145 0.0032 4.6
factors 0.0149 0.0039 3.8
graph 0.0090 0.0021 4.3
mean correlation between scores
ridge LightGBM factors graph
ridge 1.00 0.63 0.68 -0.0
LightGBM 0.63 1.00 0.44 0.0
factors 0.68 0.44 1.00 0.0
graph -0.00 0.00 0.00 1.0
correlation between the weekly IC series
ridge LightGBM factors graph
ridge 1.00 0.84 0.46 -0.00
LightGBM 0.84 1.00 0.36 -0.02
factors 0.46 0.36 1.00 0.11
graph -0.00 -0.02 0.11 1.00
```
The ridge gives the IC of the README (0.0185, t = 4.9). LightGBM and the fixed factors come in below it, and the
graph, with half the IC, has a similar t because its weekly IC is more stable. What matters is in the correlations: the
three signals that come from the features are very similar (scores between 0.44 and 0.68; IC series of ridge and
LightGBM, 0.84), and the graph resembles none of them (0.00).
## The mean and its predicted IC
```python
combos = {"ridge + LightGBM + factors": ["ridge", "LightGBM",
"factors"],
"ridge + graph": ["ridge", "graph"],
"all four": nombres}
filas = {}
for nombre, ks in combos.items():
idx = [nombres.index(k) for k in ks]
predicho = (np.mean([ics[k].mean() for k in ks])
/ np.sqrt(C[np.ix_(idx, idx)].mean()))
ic_m = ic_semanal(sum(Z[k] for k in ks) / len(ks))
dif = ic_m - ics["ridge"]
filas[nombre] = {**resumen(ic_m),
"predicted IC": round(predicho, 4),
"diff. vs ridge": round(dif.mean(), 4),
"t diff.": round(dif.mean() / dif.std(ddof=1)
* np.sqrt(len(dif)), 1)}
pd.DataFrame(filas).T
```
```output
IC standard error t predicted IC diff. vs ridge t diff.
ridge + LightGBM + factors 0.0186 0.0036 5.2 0.0188 0.0001 0.1
ridge + graph 0.0194 0.0030 6.4 0.0194 0.0010 0.5
all four 0.0205 0.0034 6.0 0.0207 0.0020 1.1
```
The formula is right to within a few ten-thousandths in all three means. And it explains the results:
- **Three signals from the same features**: high mean correlation, so the denominator barely drops below 1 and the
mean of their ICs (below the ridge) stays almost the same. The mean does not improve on the ridge.
- **Ridge + graph**: zero correlation. The mean of their ICs is low (0.0137), but it is divided by $\sqrt{0.5}$ and
rises above the ridge. The most visible change is the t: from 4.9 to 6.4.
- **All four**: 0.0205 with t = 6.0, the best IC in the table.
The t rises much more than the IC because diversification acts mainly on the **week-by-week error**: the ridge's bad
weeks are not the graph's, and the weekly IC series of the mean is more stable. The paired difference in mean IC
against the ridge, on the other hand, has t ≈ 1: the mean IC improves a little and the stability improves a lot. For a
portfolio, the second is what counts most (lesson 5.1.3: the Sharpe of a strategy is, roughly, the IC times the square
root of the breadth, divided by the variability of the IC).
## With what weights
The simple mean gives the same weight to a signal with an IC of 0.018 as to one with 0.009. It looks improvable:
weight by past IC or learn the weights with a regression (**stacking**, Wolpert, 1992; Breiman, 1996). The honesty
condition is the usual one: the weights for week $w$ can only use weeks whose label has already ended, and with
predictions that were out of sample at the time. A year of predictions is needed to start, so the comparison runs from
2016 to 2025.
Four schemes:
| Scheme | Weights | Estimated |
|---|---|---|
| Mean | 1/4 each | nothing |
| IC weights | proportional to each signal's past mean IC (0 if negative) | 4 means |
| Nested stacking | regression of the label on the 4 signals, with past weeks | 4 coefficients |
| Hindsight | the same regression with all the weeks of 2016-2025 | 4 coefficients, **with the future** |
```python
ic_w = {k: np.full(nW, np.nan) for k in nombres}
for k in nombres:
ic_w[k][evaluables] = ics[k]
por_ic = np.full((nW, nA), np.nan)
stack = np.full((nW, nA), np.nan)
pesos_stack = {}
for w in pru:
pasado = evaluables[evaluables <= ultima_conocida(w)]
if len(pasado) < 52:
continue
m = np.clip([np.mean(ic_w[k][pasado]) for k in nombres], 0, None)
por_ic[w] = sum(m[i] / m.sum() * Z[k][w]
for i, k in enumerate(nombres))
Xs = np.stack([Z[k][pasado].ravel() for k in nombres], axis=1)
b = np.linalg.lstsq(Xs, Yc[pasado].ravel(), rcond=None)[0]
stack[w] = sum(b[i] * Z[k][w] for i, k in enumerate(nombres))
pesos_stack[fechas[w]] = b / b.sum()
ev2 = np.array([w for w in evaluables if np.isfinite(stack[w]).all()])
Xr = np.stack([Z[k][ev2].ravel() for k in nombres], axis=1)
b_retro = np.linalg.lstsq(Xr, Yc[ev2].ravel(), rcond=None)[0]
media4 = sum(Z[k] for k in nombres) / 4
esquemas = {"ridge": Z["ridge"], "mean": media4,
"IC weights": por_ic, "nested stacking": stack,
"hindsight": sum(b_retro[i] * Z[k]
for i, k in enumerate(nombres))}
ic2 = {k: ic_semanal(p, ev2) for k, p in esquemas.items()}
tabla = pd.DataFrame({k: resumen(v) for k, v in ic2.items()}).T
for k, v in ic2.items():
dif = v - ic2["mean"]
et = dif.std(ddof=1) / np.sqrt(len(dif))
tabla.loc[k, "diff. vs mean"] = round(dif.mean(), 4)
tabla.loc[k, "t diff."] = (round(dif.mean() / et, 1)
if et > 1e-6 else np.nan)
print(f"{len(ev2)} weeks from {fechas[ev2[0]].date()}")
print(tabla.to_string())
print("hindsight weights:",
dict(zip(nombres, (b_retro / b_retro.sum()).round(2).tolist())))
ultimos = pesos_stack[fechas[ev2[-1]]].round(2).tolist()
print("stacking, last week:", dict(zip(nombres, ultimos)))
```
```output
521 weeks from 2016-01-08
IC standard error t diff. vs mean t diff.
ridge 0.0175 0.0040 4.4 -0.0021 -1.1
mean 0.0196 0.0036 5.5 0.0000 NaN
IC weights 0.0184 0.0037 4.9 -0.0012 -2.4
nested stacking 0.0178 0.0037 4.8 -0.0018 -1.3
hindsight 0.0201 0.0036 5.6 0.0005 0.4
hindsight weights: {'ridge': 0.62, 'LightGBM': -0.02, 'factors': 0.04, 'graph': 0.36}
stacking, last week: {'ridge': 0.66, 'LightGBM': 0.02, 'factors': 0.02, 'graph': 0.31}
```
*Fig. — Nested stacking weights (regression of the label on the four signals using past weeks), at the end of each quarter from 2016 to 2025.* Four quarterly lines. Ridge moves between 0.5 and 1.2; LightGBM (highlighted) has a negative weight for most of the period, down to −0.52, although its out-of-sample IC is positive; factors go from 0.6 to −0.28 and back to 0.04; the graph settles around 0.3. A dashed line marks equal weights, 0.25.
The simple mean beats the two schemes that estimate weights from the past, and ties with the weights chosen by looking
at the future, which are the best possible linear combination in this sample. It is the **forecast combination
puzzle**: in studies with macroeconomic data, the simple mean beats “optimal” combinations again and again. Smith and
Wallis (2009) give the explanation: the optimal sample weights carry estimation error, and when the signals have small,
similar ICs, that error costs more than what is gained by choosing well. The figure shows the stacking weights over
time: they move with the noise of each year, and in some stretches they give negative weight to signals that are good
out of sample.
## From signal to portfolio
The IC is the metric of the signal; the portfolio also pays for turnover. A long-short decile portfolio, equally
weighted and rebalanced every week, with 10 bp per side on turnover (as in the README):
```python
def larga_corta(p, semanas=ev2, coste=0.001):
rend, rot, previo = [], [], np.zeros(nA)
for w in semanas:
rk = rankdata(p[w])
q = np.where(rk > 0.9 * nA, 1.0, 0.0)
q = q / q.sum() - np.where(rk <= 0.1 * nA, 1.0, 0.0) \
/ (rk <= 0.1 * nA).sum()
rend.append(q @ R[w])
rot.append(np.abs(q - previo).sum())
previo = q
rend, rot = np.array(rend), np.array(rot)
neto = rend - coste * rot
return {"annual ret. %": 5200 * rend.mean(),
"gross Sharpe": np.sqrt(52) * rend.mean() / rend.std(),
"weekly turnover %": 100 * rot[1:].mean(),
"net Sharpe": np.sqrt(52) * neto.mean() / neto.std()}
pd.DataFrame({k: larga_corta(Z[k] if k in Z else media4)
for k in ("ridge", "graph", "mean of the four")}
).T.round(2)
```
```output
annual ret. % gross Sharpe weekly turnover % net Sharpe
ridge 10.77 1.13 262.72 -0.30
graph 6.06 1.06 322.34 -1.87
mean of the four 10.04 1.18 272.89 -0.49
```
Combining does not fix costs. The mean of the four signals has a gross Sharpe similar to the ridge's (1.18 against
1.13) and turns over somewhat more (273% against 263% a week), so net of costs it still does not make money (−0.49
against −0.30). The graph signal alone, with a gross Sharpe of 1.06, is the one that comes out worst net (−1.87): it
changes sign every week with the customers' return and turns over 322%. In all three cases, the turnover of a weekly cross-sectional signal eats up
an edge of this size (lesson 5.1.3). Combining improves the ingredient; the portfolio recipe (rebalancing frequency,
no-trade bands, cost per trade) is another lesson.
## Synthetic data: GANs and diffusion
The other half of the section is generating data. Lesson 2.5.7 used explicit-model generators (GBM, GARCH, bootstrap)
for stress tests, and all the data in this course come from models with published parameters. Deep generators promise
something else: learning the distribution of the series directly from the data, without writing down the model.
- **GANs** (Goodfellow et al., 2014): a generator turns noise into series and a discriminator tries to tell them apart
from the real ones; they are trained against each other. In finance, **Quant GANs** (Wiese et al., 2020) uses
temporal convolutional networks in both and claims to reproduce volatility clustering, the leverage effect and
autocorrelations; **TimeGAN** (Yoon et al., 2019) adds a supervised loss on the temporal dynamics.
- **Diffusion** (Ho et al., 2020): the model learns to remove noise from data to which noise has been added step by
step, and generates from pure noise. Takahashi and Mizuno (2024) apply diffusion models to series of prices, volumes
and bid-ask spreads converted into images with a wavelet transform and claim to reproduce fat tails, volatility
clustering and seasonality; CoFinDiff (Tanaka et al., 2025) conditions the generation on a trend and a realized
volatility.
What these papers claim is checked with three families of tests, which are also the ones highlighted by the review of
Assefa et al. (2020) on synthetic data in finance:
| Test | Question | How |
|---|---|---|
| Stylized facts | Does it reproduce tails, volatility clustering, near-zero autocorrelation of returns? | kurtosis, extreme quantiles, autocorrelation of r and of \|r\| |
| Memorization | Does it copy the training sample? | distance from each generated series to the nearest training one, against that of new real data |
| External discriminator | Can it be told apart from the real data? | a classifier outside training; AUC 0.5 = indistinguishable |
## A small GAN against the bootstrap
To get hands-on, a minimal GAN in PyTorch that generates windows of 20 sessions of returns of the `indice` asset in
`vol5_diario.parquet` (the level's volatility lab: GARCH-HAR with t tails and leverage effect). It is trained on
2008-2019 (3,002 overlapping, standardized windows) and compared with the 2020-2025 data, with normal noise and with
the stationary bootstrap of lesson 2.5.7 (blocks of 10 sessions on average). A real generator would train much longer
and with temporal networks; this one takes a few seconds on CPU.
```python
import torch
from scipy.stats import kurtosis
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
vol = pd.read_parquet("vol5_diario.parquet")
r = vol[vol["activo"] == "indice"].set_index("fecha")["rendimiento"]
L, sd = 20, r[:"2019-12-31"].std()
ventanas = lambda x: np.lib.stride_tricks.sliding_window_view(x, L)
V_ent = ventanas(r[:"2019-12-31"].to_numpy() / sd)
V_pru = ventanas(r["2020-01-01":].to_numpy() / sd)
def red(entrada, salida):
return torch.nn.Sequential(
torch.nn.Linear(entrada, 64), torch.nn.LeakyReLU(0.2),
torch.nn.Linear(64, 64), torch.nn.LeakyReLU(0.2),
torch.nn.Linear(64, salida))
gen, disc = red(16, L), red(L, 1)
o_g = torch.optim.Adam(gen.parameters(), lr=2e-4, betas=(0.5, 0.999))
o_d = torch.optim.Adam(disc.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = torch.nn.BCEWithLogitsLoss()
reales = torch.tensor(V_ent, dtype=torch.float32)
azar = torch.Generator().manual_seed(0)
uno, cero = torch.ones(128, 1), torch.zeros(128, 1)
for paso in range(3000):
lote = reales[torch.randint(0, len(reales), (128,), generator=azar)]
falsas = gen(torch.randn(128, 16, generator=azar))
o_d.zero_grad()
p_d = bce(disc(lote), uno) + bce(disc(falsas.detach()), cero)
p_d.backward()
o_d.step()
o_g.zero_grad()
p_g = bce(disc(falsas), uno)
p_g.backward()
o_g.step()
with torch.no_grad():
V_gan = gen(torch.randn(3000, 16, generator=azar)).numpy()
V_gan = V_gan.astype(float)
print(f"windows: training {V_ent.shape}, test {V_pru.shape}, "
f"GAN {V_gan.shape}")
print(f"final losses: discriminator {p_d.item():.2f}, "
f"generator {p_g.item():.2f}")
```
```output
windows: training (3002, 20), test (1489, 20), GAN (3000, 20)
final losses: discriminator 1.20, generator 0.80
```
The final losses do not say whether the GAN is good: in a GAN, the two networks can be in equilibrium and the
generator can produce series that look nothing like the real ones in what matters. It has to be measured.
```python
rng = np.random.default_rng(0)
def bootstrap_estacionario(x, n, media=10):
out = np.empty((n, L))
for i in range(n):
j, serie = rng.integers(len(x)), []
while len(serie) < L:
serie.append(x[j])
j = (rng.integers(len(x)) if rng.random() < 1 / media
else (j + 1) % len(x))
out[i] = serie
return out
V_boot = bootstrap_estacionario(r[:"2019-12-31"].to_numpy() / sd, 3000)
V_norm = rng.standard_normal((3000, L))
def hechos(V):
def ac(Y, k):
return np.corrcoef(Y[:, :-k].ravel(), Y[:, k:].ravel())[0, 1]
return {"kurtosis": kurtosis(V.ravel()),
"0.5% quantile": np.quantile(V.ravel(), 0.005),
"AC(1) of r": ac(V, 1),
"AC(1) of |r|": ac(np.abs(V), 1),
"AC(5) of |r|": ac(np.abs(V), 5)}
conjuntos = {"real 2008-2019": V_ent, "real 2020-2025": V_pru,
"GAN": V_gan, "normal": V_norm, "bootstrap": V_boot}
pd.DataFrame({k: hechos(v) for k, v in conjuntos.items()}).T.round(3)
```
```output
kurtosis 0.5% quantile AC(1) of r AC(1) of |r| AC(5) of |r|
real 2008-2019 3.793 -3.357 -0.026 0.088 0.028
real 2020-2025 1.865 -3.449 0.055 0.108 0.068
GAN 0.616 -2.709 -0.240 0.092 0.125
normal 0.023 -2.575 -0.002 0.005 -0.010
bootstrap 3.654 -3.357 -0.024 0.080 0.009
```
*Fig. — Two stylised facts of 20-session windows of the indice asset: real, generated by a small GAN, normal and by stationary bootstrap.* Two bar panels. Excess kurtosis: real 2008-2019 3.79; real 2020-2025 1.87; GAN 0.62 (highlighted); normal 0.02; bootstrap 3.65. Lag-1 autocorrelation of returns: real 2008-2019 −0.03; real 2020-2025 0.06; GAN −0.24 (highlighted); normal 0.00; bootstrap −0.02. The GAN has no tails and invents a negative autocorrelation.
The GAN learned the scale of the returns and the lag-1 autocorrelation of |r| (0.092 against 0.088), but it fails at
the rest: its kurtosis is a sixth of the real one (0.62 against 3.79: almost normal tails, with a 0.5% quantile of
−2.71 against −3.36), it exaggerates the memory of |r| at five sessions (0.125 against 0.028) and it invents a lag-1
autocorrelation of returns of −0.24 that the real series does not have (−0.03). The bootstrap, which only reshuffles
blocks of the history, reproduces the tails and the autocorrelation of returns by construction and keeps part of the
clustering.
```python
from sklearn.neighbors import NearestNeighbors
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
vecino = NearestNeighbors(n_neighbors=1).fit(V_ent)
def resumen_ventana(V):
a = np.abs(V)
ac = [np.corrcoef(x[:-1], x[1:])[0, 1] for x in a]
return np.c_[V.std(1), kurtosis(V, axis=1), ac, a.max(1), V.sum(1)]
X_real = resumen_ventana(V_pru[::5]) # 298 real windows
filas = {}
for k in ("real 2020-2025", "GAN", "normal", "bootstrap"):
V = conjuntos[k]
fila = {"nearest-neighbor distance (median)":
np.median(vecino.kneighbors(V[:1000])[0])}
if k != "real 2020-2025":
X_s = resumen_ventana(V[:len(X_real)])
X_c = np.r_[X_real, X_s]
y_c = np.r_[np.zeros(len(X_real)), np.ones(len(X_s))]
fila["discriminator AUC"] = cross_val_score(
LogisticRegression(max_iter=1000), X_c, y_c, cv=5,
scoring="roc_auc").mean()
filas[k] = fila
pd.DataFrame(filas).T.round(3)
```
```output
nearest-neighbor distance (median) discriminator AUC
real 2020-2025 2.963 NaN
GAN 2.969 0.754
normal 3.288 0.627
bootstrap 2.671 0.457
```
The distance from the GAN's windows to the nearest training window is that of new real data: it does not copy the
sample (memorization is the risk of large generators with little data, and also a privacy problem when the data belong
to clients). The external discriminator, a logistic regression with five summaries of each window, tells the GAN's
windows apart from the real ones with an AUC of 0.75, better than the normal ones (0.63); the bootstrap's (0.46) it
cannot tell apart. A deep generator that does not beat a bootstrap on these tests adds nothing to a stress test.
## What synthetic data cannot do
| Use | Does it make sense? | Why |
|---|---|---|
| Stress tests and scenarios | yes, with evaluated generators | more paths than history, with controlled tails and clustering (lesson 2.5.7) |
| Testing methods with a known answer | yes | it is what this course does: explicit models with planted effects |
| Sharing data without exposing clients | sometimes | if it is shown that there is no memorization |
| Augmenting data to train forecasting models | with great care | the generator comes from the same history: it adds no information about the future; any improvement is regularization and has to be compared with regularizing directly |
| Source of signal | no | a generator reproduces the distribution it saw, noise included; the edge a model “discovers” in synthetic data is the generator's |
Deep generators are one more modeling tool, with the same underlying problem as any market model: a market is a single
history that does not repeat, and what is not in it (the next crisis, with its new shape) is not in a generator trained
on it either.
## Exercises
1. With two uncorrelated signals, the optimal combination weights each one in proportion to its IC and reaches an IC of
$\sqrt{\rho_1^2 + \rho_2^2}$. Compute that ceiling for the ridge and the graph and compare it with the simple mean
of the two and with stacking of just those two signals.
**Solution.**
```python
r1, r2 = ics["ridge"].mean(), ics["graph"].mean()
techo = np.sqrt(r1 ** 2 + r2 ** 2)
optima = ic_semanal(r1 * Z["ridge"] + r2 * Z["graph"])
media2 = ic_semanal((Z["ridge"] + Z["graph"]) / 2)
st2 = np.full((nW, nA), np.nan)
for w in ev2:
pasado = evaluables[evaluables <= ultima_conocida(w)]
Xs = np.stack([Z[k][pasado].ravel() for k in ("ridge", "graph")], 1)
b = np.linalg.lstsq(Xs, Yc[pasado].ravel(), rcond=None)[0]
st2[w] = b[0] * Z["ridge"][w] + b[1] * Z["graph"][w]
print(f"theoretical ceiling {techo:.4f} | IC weights (known at the end) "
f"{optima.mean():.4f} | mean {media2.mean():.4f}")
m2 = ic_semanal((Z["ridge"] + Z["graph"]) / 2, ev2).mean()
print(f"in 2016-2025: mean {m2:.4f} | stacking "
f"{ic_semanal(st2, ev2).mean():.4f}")
```
```output
theoretical ceiling 0.0205 | IC weights (known at the end) 0.0205 | mean 0.0194
in 2016-2025: mean 0.0193 | stacking 0.0193
```
The weights proportional to the IC, computed over the whole period (in hindsight), reach the ceiling (0.0205); the
simple mean stays at 0.0194, because it gives too much weight to the graph. But when the weights are truly estimated
from the past (nested stacking, 2016-2025), they tie with the mean: 0.0193 both. The theoretical advantage of weighting
well is eaten up by the error of estimating the weights, even with one signal clearly weaker than the other. The
practical rule: equal weights unless there are strong reasons (and a lot of history) for something else.
2. Train the GAN for twice as many steps (6,000) with the same seed. Do the kurtosis, the lag-1 autocorrelation and the
discriminator AUC improve?
**Solution.**
```python
torch.manual_seed(0)
gen, disc = red(16, L), red(L, 1)
o_g = torch.optim.Adam(gen.parameters(), lr=2e-4, betas=(0.5, 0.999))
o_d = torch.optim.Adam(disc.parameters(), lr=2e-4, betas=(0.5, 0.999))
azar = torch.Generator().manual_seed(0)
for paso in range(6000):
lote = reales[torch.randint(0, len(reales), (128,), generator=azar)]
falsas = gen(torch.randn(128, 16, generator=azar))
o_d.zero_grad()
(bce(disc(lote), uno) + bce(disc(falsas.detach()), cero)).backward()
o_d.step()
o_g.zero_grad()
bce(disc(falsas), uno).backward()
o_g.step()
with torch.no_grad():
V_gan2 = gen(torch.randn(3000, 16, generator=azar)).numpy()
V_gan2 = V_gan2.astype(float)
X_s = resumen_ventana(V_gan2[:len(X_real)])
auc = cross_val_score(LogisticRegression(max_iter=1000),
np.r_[X_real, X_s],
np.r_[np.zeros(len(X_real)), np.ones(len(X_s))],
cv=5, scoring="roc_auc").mean()
print({k: round(float(v), 3) for k, v in hechos(V_gan2).items()},
"| AUC", round(float(auc), 3))
```
```output
{'kurtosis': 1.076, '0.5% quantile': -2.428, 'AC(1) of r': -0.15, 'AC(1) of |r|': 0.088, 'AC(5) of |r|': 0.044} | AUC 0.773
```
With 6,000 steps the kurtosis rises somewhat (1.08) and the spurious autocorrelation falls (−0.15), but the
discriminator tells it apart better than before (AUC 0.77 against 0.75). Training longer moves the figures, but neither
monotonically nor toward what is sought: a GAN does not minimize a loss that
measures the stylized facts, and its training oscillates. That is why serious work chooses the architecture (temporal
networks that can represent volatility memory, tails in the input noise) and evaluates with tests fixed in advance,
not with the discriminator loss.
3. A colleague proposes training the panel's signal model with ten times more weeks “generated with diffusion from the
real panel”. What validation would you ask for before accepting it?
**Solution.**
First, that the generator is fitted only on the training data of each walk-forward step (if it sees the test period,
it is a leak through the back door). Second, a paired out-of-sample comparison between the model trained on real data
and the one trained on real plus synthetic data, with the same walk-forward and their IC difference with its standard
error. Third, a comparison with regularizing directly (more penalty, fewer trees, seed ensembles): if the synthetic data
help, they most likely do so as regularization, and then the cheap option wins. And fourth, the generator's
memorization tests: ten times more weeks that are near copies of the real ones only multiply the overfitting.
## Self-assessment
1. Ridge, LightGBM and a fixed factor combination, all with the same features, have ICs of 0.0185, 0.0145 and 0.0149, with correlations between their scores of 0.44 to 0.68. Their mean has an IC of 0.0186. Why does it not improve on the ridge?
- A. Because a mean of signals never improves on the best one
- B. Because LightGBM is badly trained
- C. Because the three models extract the same information from the same features: their errors are highly correlated and the mean dilutes the best one while diversifying almost nothing
- D. Because the mean should be multiplied by three
*Answer C: Because the three models extract the same information from the same features: their errors are highly correlated and the mean dilutes the best one while diversifying almost nothing.* The IC of a mean of standardized signals is the mean of the ICs divided by the square root of the mean correlation between them. With high correlations the denominator barely drops below 1 and offsets just what the mean of the ICs (0.0160) loses against the best one. The formula predicts 0.0188 and 0.0186 is measured.
2. The customer-supplier graph signal has an IC of only 0.009, but zero correlation with the ridge. What happens when it is averaged with it?
- A. The IC of the mean rises and, above all, becomes more stable: the week-by-week errors of the two signals do not coincide and partly offset each other
- B. The IC falls because the graph signal is weaker
- C. Nothing changes: only the best signal matters
- D. The IC rises but its t falls
*Answer A: The IC of the mean rises and, above all, becomes more stable: the week-by-week errors of the two signals do not coincide and partly offset each other.* With zero correlation, the IC of the mean is (0.0185 + 0.0090)/2/√0.5 ≈ 0.0194, more than the ridge alone. And the weekly IC series of the mean is less volatile, so its t rises much more than its mean: that is error diversification.
3. Nested stacking (a regression of the label on the signals, estimated only with past weeks) gives an IC of 0.0178 and the simple mean of the four signals, 0.0196. What is this?
- A. A bug in the stacking
- B. The forecast combination puzzle: with little signal, the error of estimating the weights costs more than what is gained by choosing them, and equal weights usually win
- C. Proof that stacking is always worse
- D. That the simple mean uses future information
*Answer B: The forecast combination puzzle: with little signal, the error of estimating the weights costs more than what is gained by choosing them, and equal weights usually win.* Smith and Wallis (2009) explain it this way: the optimal sample weights are estimated with a lot of noise when the signals have similar, small ICs. Equal weights are not estimated and do not carry that error.
4. A GAN trained on 20-day windows of returns has a kurtosis of 0.6 (the real series, 3.8), a lag-1 autocorrelation of −0.24 (the real one, −0.03) and an external discriminator tells it apart from the real data with an AUC of 0.75. The stationary bootstrap gives 0.46. What is the conclusion?
- A. That the GAN is better because it generates new data
- B. That AUC is not a good measure
- C. That the GAN should be used to train the models
- D. That this GAN does not reproduce the tails and invents a dependence that does not exist; for stress tests, the bootstrap of lesson 2.5.7 is better and much cheaper
*Answer D: That this GAN does not reproduce the tails and invents a dependence that does not exist; for stress tests, the bootstrap of lesson 2.5.7 is better and much cheaper.* A generator is evaluated by what it reproduces and by what it invents. This small GAN fails at both; those in the literature (Quant GANs, diffusion) claim to do better, but they are evaluated with the same tests.
5. A vendor sells synthetic data generated with diffusion from the history of a market and claims that training on it improves forecasting models. What is the main objection?
- A. That diffusion is slower than a GAN
- B. That a generator trained on a single history contains no more information about the future than that history: it can help regularize or run stress tests, but it does not create signal, and it can copy the noise of the sample
- C. That synthetic data is always illegal
- D. That it needs a GPU
*Answer B: That a generator trained on a single history contains no more information about the future than that history: it can help regularize or run stress tests, but it does not create signal, and it can copy the noise of the sample.* Augmenting data with a generator fitted to the same sample reuses the same information. If the model improves in an honest validation, the improvement comes from regularization, and it has to be compared with regularizing directly.
## Key points
- The IC of a mean of k standardized signals is the mean of their ICs divided by the square root of their mean correlation. It works on the data: it predicts 0.0207 for the mean of the four signals and 0.0205 is measured.
- Signals extracted from the same features (ridge, LightGBM, fixed factors) correlate between 0.44 and 0.68 and their mean does not improve on the ridge. A signal from another source (the graph, correlation 0) does improve the mean and, above all, its stability.
- Equal weights beat estimated ones (past IC, nested stacking) and tie with weights chosen in hindsight: the forecast combination puzzle. Hindsight weights are not a result, they are a leak.
- Combining stabilizes the IC more than it raises it: the IC t of the mean of the four signals is 5.5 against 4.4 for the ridge alone, with a paired difference in mean IC that is not conclusive.
- GANs and diffusion promise series with tails, volatility clustering and leverage. They are evaluated with stylized facts, memorization and an external discriminator. A small GAN fails where a bootstrap succeeds.
- A generator fitted to one history does not create information about the future: it serves for stress tests, privacy and regularization, not as a source of signal.
## Resources
### Books
- [Bates, J. M. and Granger, C. W. J. (1969): The Combination of Forecasts. Journal of the Operational Research Society 20(4), 451-468](https://doi.org/10.1057/jors.1969.103) · doi.org
- [Smith, J. and Wallis, K. F. (2009): A Simple Explanation of the Forecast Combination Puzzle. Oxford Bulletin of Economics and Statistics 71(3), 331-355](https://doi.org/10.1111/j.1468-0084.2008.00541.x) · doi.org
- [Wolpert, D. H. (1992): Stacked generalization. Neural Networks 5(2), 241-259](https://doi.org/10.1016/S0893-6080(05)80023-1) · doi.org
- [Wiese, M. et al. (2020): Quant GANs: deep generation of financial time series. Quantitative Finance 20(9), 1419-1440](https://doi.org/10.1080/14697688.2020.1730426) · doi.org
- [Takahashi, T. and Mizuno, T. (2024): Generation of synthetic financial time series by diffusion models (arXiv:2410.18897)](https://arxiv.org/abs/2410.18897) · arxiv.org
- [Assefa, S. A. et al. (2020): Generating synthetic data in finance: opportunities, challenges and pitfalls. ICAIF 2020](https://doi.org/10.1145/3383455.3422554) · doi.org
### Data
- [Level 5 data: weekly panel features (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet
Previous lesson: [5.10.2 · Data drift and concept drift: detecting them and what to do](https://pythonparatrading.com/en/python-trading-course/level-5/data-drift-concept-drift/)