---
title: "5.7.1 PyTorch: tensors, autograd and the training loop"
description: "Tensors, autograd checked with finite differences, Dataset and DataLoader with time splits, a clean loop, determinism, torch.compile and Lightning."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/pytorch-tensors-autograd-training-loop/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/pytorch-tensors-autograd-training-loop.png"
---
# 5.7.1 · PyTorch: tensors, autograd and the training loop
> Keras hides the training loop behind `fit`; PyTorch leaves it in plain sight. This lesson writes it piece by piece on a problem with a known answer, tomorrow's realized volatility: tensors, gradients checked with finite differences, data split by date, a loop with early stopping and the same network in Lightning, with bit-for-bit identical results.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.7 · Deep learning with PyTorch
- Duration: 75 min
- Prerequisites: [5.6.2 · An MLP on a stock panel: regularize and compare](https://pythonparatrading.com/en/python-trading-course/level-5/mlp-for-tabular-financial-data/), [5.4.2 · Purged k-fold with embargo and CPCV for ML models](https://pythonparatrading.com/en/python-trading-course/level-5/purged-k-fold-embargo-cpcv/), [2.1.4 · Volatility models: ARCH, GARCH, EGARCH and GJR with the arch library](https://pythonparatrading.com/en/python-trading-course/level-2/volatility-models-arch-garch-egarch-gjr/)
## By the end of this lesson
- Handle tensors without surprises: memory shared with NumPy, `dtype` (`float32` versus `float64`), broadcasting and device.
- Compute gradients with autograd and check them against the analytical derivative and against finite differences, knowing how to choose the step at each precision.
- Build a `Dataset` and a `DataLoader` with date splits, purging at the boundary and normalization with training statistics.
- Write a clean training loop (`train`/`eval` modes, `zero_grad`, early stopping) that is deterministic, and compare it with the closed-form solution and with a baseline.
- Place GPU, MPS, `torch.compile` and Lightning: what they bring, what they cost and how to reproduce with Lightning the same result as the handwritten loop.
## From Keras's `fit` to a loop in plain sight
In module 5.6, Keras trains with one call: `model.fit(x, y, epochs=…)`. Inside there is a loop that requests batches,
computes the loss, differentiates, updates the weights and evaluates. PyTorch does not hide that loop: you write it.
It is more code, and it is the reason almost all deep learning research is published in PyTorch: when something
fails, the failure is in lines you can read.
| Piece | Keras 3 (5.6) | PyTorch | Lightning |
|---|---|---|---|
| Data | arrays or `PyDataset` | `Dataset` + `DataLoader` | the same `DataLoader`s |
| Model | `keras.Model` | `nn.Module` | `LightningModule` (wraps one) |
| Gradients | hidden in `fit` | `loss.backward()` | hidden in the `Trainer` |
| Loop | `fit` | handwritten | `Trainer.fit` |
| Device | backend and configuration | explicit `.to(device)` | `accelerator=` |
The problem of the lesson is the level 5 volatility one: predict the logarithm of tomorrow's **realized variance** for
six fictitious assets (`vol5_diario.parquet`) with the features of Corsi's (2009) HAR model: today's RV, its 5-session
mean and its 22-session mean, plus the negative part of today's return. The data are synthetic and the truth is
known: the generator is a HAR on the logarithm of integrated variance, so a linear HAR is almost optimal and serves
as the yardstick. A network that beat it by a wide margin would be measuring a mistake.
```python
import copy
import numpy as np
import pandas as pd
import torch
from torch import nn
from torch.utils.data import DataLoader, Dataset
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
v = pd.read_parquet("vol5_diario.parquet")
v["x"] = np.log(v["rv_5min"])
def har(g):
"""HAR features of one asset and target: tomorrow's log RV."""
x = g["x"]
return pd.DataFrame({
"fecha": g["fecha"], "activo": g["activo"],
"d": x, "w": x.rolling(5).mean(), "m": x.rolling(22).mean(),
"neg": np.minimum(g["rendimiento"], 0),
"y": x.shift(-1), "fecha_y": g["fecha"].shift(-1)})
h = pd.concat([har(g) for _, g in v.groupby("activo")]).dropna()
h["tramo"] = np.select(
[h["fecha_y"] < "2016-01-01",
(h["fecha"] >= "2016-01-01") & (h["fecha_y"] < "2018-01-01"),
h["fecha"] >= "2018-01-01"],
["ent", "val", "prueba"], "purga")
ent = h["tramo"] == "ent"
centro = h[ent].groupby("activo")["d"].mean()
escala = h[ent].groupby("activo")["neg"].std()
for c in ["d", "w", "m", "y"]:
h[c] = h[c] - h["activo"].map(centro)
h["neg"] = h["neg"] / h["activo"].map(escala)
print(h["tramo"].value_counts().to_string())
print(h.loc[h["tramo"] == "purga", ["fecha", "fecha_y"]]
.drop_duplicates().to_string(index=False))
```
```output
tramo
prueba 12060
ent 11958
val 3012
purga 12
fecha fecha_y
2015-12-31 2016-01-04
2017-12-29 2018-01-02
```
Three decisions that recur throughout the module (in the code, `ent`, `val` and `prueba` are the training, validation
and test segments, and `purga` the purged rows):
- **The split is made by the label date.** Training: targets before 2016; validation: 2016 and 2017; test: 2018
onward. Rows whose target crosses a boundary (the one of December 31, 2015 points to the RV of January 4, 2016) are
**purged**, as in lesson 5.4.2 with longer labels.
- **The statistics come from training.** The center of each asset (the mean of its log RV) and the scale of negative
returns are computed with 2008-2015 and applied unchanged to validation and test. Computing them with the whole
sample puts the future into the past (lesson 5.2.4).
- **One global model.** The six assets go together, each centered on its own level: 12,000 training rows instead of
2,000. The level shifts in the sample (the bond in 2022, crypto in 2019) are outside the center computed with
2008-2015: the model has to follow them through the persistence of the features themselves.
## Tensors
A tensor is an n-dimensional array with three attributes that matter: `dtype`, `shape` and `device`. And, on request,
a history of operations for differentiation. Almost the whole API imitates NumPy; the differences are in the details.
```python
a = h[["d", "w"]].to_numpy()[:3].copy()
t = torch.from_numpy(a) # shares memory with `a`
print(t.dtype, tuple(t.shape))
t[0, 0] = 9.0
print("a[0, 0] after changing t:", a[0, 0])
c = torch.tensor(a) # copy
c[0, 0] = -1.0
print("a[0, 0] after changing c:", a[0, 0])
print("from Python floats:", torch.tensor([0.5, 1.0]).dtype)
x = torch.arange(6, dtype=torch.float32).reshape(3, 2)
print(x - x.mean(dim=0)) # (3, 2) - (2,)
print(x.T.is_contiguous(), x.device)
```
```output
torch.float64 (3, 2)
a[0, 0] after changing t: 9.0
a[0, 0] after changing c: 9.0
from Python floats: torch.float32
tensor([[-2., -2.],
[ 0., 0.],
[ 2., 2.]])
False cpu
```
- `torch.from_numpy` does not copy: changing the tensor changes the array. `torch.tensor` does copy.
- A NumPy array is `float64`; a tensor created from Python floats is `float32`, PyTorch's default `dtype` and that of
the layers. Mixing them is not converted silently:
```python raises
capa = nn.Linear(2, 1) # float32 weights
capa(torch.from_numpy(a)) # float64 input
```
```output
RuntimeError: mat1 and mat2 must have the same dtype, but got Double and Float
```
- Broadcasting follows NumPy's rules: `(3, 2) - (2,)` subtracts the mean of each column.
- `x.T` is a **view** with different memory strides (not contiguous): it copies nothing. Some operations (`view`)
reject it; `reshape` or `.contiguous()` copy when needed.
- `x.device` says where the tensor lives. Two tensors on different devices cannot be operated on together.
## Autograd and how to check it
When a tensor has `requires_grad=True`, PyTorch records each operation that uses it in a directed **graph**. When
`backward()` is called on a scalar, it walks that graph backward with the chain rule and leaves in each leaf its
derivative in `.grad`. It is reverse-mode automatic differentiation: the cost is of the order of one more evaluation
of the function, whether it has one parameter or a million.
To see it with one of the losses used in volatility: a linear model of the logarithm of variance, $f = X w$,
evaluated with the **QLIKE** (the loss of the forecast table in the data README), which up to constants is
$$
L(w) = \frac{1}{n}\sum_i \left( RV_i\, e^{-f_i} + f_i \right),
\qquad
\nabla L = \frac{1}{n} X^\top \left( 1 - RV \odot e^{-f} \right).
$$
```mermaid
flowchart LR
W[w leaf] --> F[f = X w]
X[X data] --> F
F --> E[RV e^-f + f]
E --> L[mean L]
L -. backward .-> W
```
```python
cols = ["d", "w", "m", "neg"]
Xq = torch.tensor(h.loc[ent, cols].to_numpy())
Xq = torch.cat([torch.ones(len(Xq), 1, dtype=Xq.dtype), Xq], dim=1)
rv = torch.exp(torch.tensor(h.loc[ent, "y"].to_numpy()))
def qlike(w):
f = Xq @ w
return (rv * torch.exp(-f) + f).mean()
w = torch.tensor([0.0, 0.4, 0.3, 0.2, -0.1], dtype=torch.float64,
requires_grad=True)
perdida = qlike(w)
perdida.backward()
with torch.no_grad():
f = Xq @ w
analitico = Xq.T @ (1 - rv * torch.exp(-f)) / len(Xq)
print("autograd :", w.grad.numpy().round(6))
print("analytical:", analitico.numpy().round(6))
print("max. difference:", f"{(w.grad - analitico).abs().max():.1e}")
```
```output
autograd : [-0.127462 0.087152 0.045058 0.02778 -0.022228]
analytical: [-0.127462 0.087152 0.045058 0.02778 -0.022228]
max. difference: 1.6e-15
```
They agree to the last decimal of `float64`. When there is no analytical derivative (a network, an unusual loss), the
check uses **central finite differences**: $\partial L/\partial w_j \approx (L(w + h e_j) - L(w - h
e_j)) / 2h$. Its error has two parts pulling in opposite directions: truncation, proportional to $h^2$, and the
rounding from subtracting two nearly equal numbers, proportional to $\varepsilon/h$, with $\varepsilon$ the machine
precision. The best step is where they cross, around $\varepsilon^{1/3}$.
```python
def dif_finitas(w0, paso, dtype):
X_, rv_ = Xq.to(dtype), rv.to(dtype)
w0 = w0.detach().to(dtype)
g = torch.zeros_like(w0)
for j in range(len(w0)):
e = torch.zeros_like(w0)
e[j] = paso
Lmas = (rv_ * torch.exp(-X_ @ (w0 + e)) + X_ @ (w0 + e)).mean()
Lmen = (rv_ * torch.exp(-X_ @ (w0 - e)) + X_ @ (w0 - e)).mean()
g[j] = (Lmas - Lmen) / (2 * paso)
return g
errores = {}
for dt in (torch.float64, torch.float32):
for k in range(1, 10):
g = dif_finitas(w, 10.0 ** -k, dt).double()
errores[(str(dt)[6:], f"1e-{k}")] = (g - w.grad).abs().max()
err = pd.Series(errores).unstack(0)
print(err.map(lambda e: f"{e:.1e}").to_string())
print("gradcheck:", torch.autograd.gradcheck(qlike, (w,)))
```
```output
float32 float64
1e-1 5.9e-03 5.9e-03
1e-2 5.5e-05 5.8e-05
1e-3 9.2e-05 5.8e-07
1e-4 9.5e-04 5.8e-09
1e-5 9.7e-03 6.5e-11
1e-6 8.7e-02 1.1e-10
1e-7 1.3e-01 1.2e-09
1e-8 1.3e-01 8.9e-09
1e-9 1.3e-01 1.1e-07
gradcheck: True
```
*Fig. — Maximum error of the QLIKE gradient computed with central finite differences against autograd, by step size and precision.* Two curves on a log-log scale. With large steps both coincide and the error falls two decades per decade of step (truncation). In float32 the minimum is 5.5e-5 at h = 1e-2 and then the error rises to 0.13 at h = 1e-7. In float64 the minimum (highlighted) is 6.5e-11 at h = 1e-5, and with smaller steps rounding pushes it up to 1.1e-7 at h = 1e-9.
The figure puts the table on a log scale. In `float64` the error falls with $h^2$ (each decade of step, two decades
of error) down to $h = 10^{-5}$, where it reaches 6.5·10⁻¹¹; with smaller steps, rounding pushes it back up. In
`float32` the minimum comes much earlier, at $h = 10^{-2}$ with an error of 5.5·10⁻⁵, and at $h = 10^{-6}$ the error
(0.087) is already the size of the gradient itself: the estimate is garbage. Hence the rule of thumb: gradient checks
are done in `float64`, which is what `torch.autograd.gradcheck` does (and why it requires double-precision inputs).
## Dataset and DataLoader with time splits
A `Dataset` answers two questions: how many examples there are (`__len__`) and which one is the i-th
(`__getitem__`). A `DataLoader` groups them into batches, shuffles them if asked and can load them in parallel
processes. The normalization of the features (mean and standard deviation of each column) comes, once again, from
training.
```python
FEATS = ["d", "w", "m", "neg"]
media = h.loc[ent, FEATS].mean()
desv = h.loc[ent, FEATS].std()
class SerieVol(Dataset):
"""Features standardized with training and tomorrow's log RV."""
def __init__(self, df):
z = (df[FEATS] - media) / desv
self.X = torch.tensor(z.to_numpy(), dtype=torch.float32)
self.y = torch.tensor(df["y"].to_numpy(), dtype=torch.float32)
self.fecha_y = df["fecha_y"].to_numpy()
def __len__(self):
return len(self.y)
def __getitem__(self, i):
return self.X[i], self.y[i]
ds = {k: SerieVol(h[h["tramo"] == k]) for k in ("ent", "val", "prueba")}
print({k: len(d) for k, d in ds.items()})
print("last target in ent:",
ds["ent"].fecha_y.max().astype("datetime64[D]"))
print("first target in val:",
ds["val"].fecha_y.min().astype("datetime64[D]"))
def cargador(semilla=0):
return DataLoader(ds["ent"], batch_size=256, shuffle=True,
generator=torch.Generator().manual_seed(semilla))
xb, yb = next(iter(cargador()))
print(tuple(xb.shape), tuple(yb.shape), xb.dtype)
```
```output
{'ent': 11958, 'val': 3012, 'prueba': 12060}
last target in ent: 2015-12-31
first target in val: 2016-01-05
(256, 4) (256,) torch.float32
```
Shuffling **within** training is correct: the rows are already split by date and each one carries only information
from its past, so the batch order leaks nothing; shuffling helps stochastic gradient descent. What is wrong is
shuffling **before** splitting (`random_split` over the whole sample): it is the random k-fold of lesson 5.4.1, with
almost identical neighbors on both sides of the split.
The dedicated `generator` is not a detail: it makes the batch order depend only on its seed, not on how many random
numbers the rest of the program has consumed before. The Lightning section shows why that matters.
## nn.Module and a clean training loop
A model is a subclass of `nn.Module`: the layers are declared in `__init__` (their parameters are registered) and
`forward` says how they combine. The loop is always the same:
```mermaid
flowchart TB
A[Batch x, y] --> B[zero_grad]
B --> C[forward]
C --> D[loss]
D --> E[backward]
E --> F[step]
F --> A
F --> G[val with no_grad]
G --> H[best state_dict]
```
```python
class Lineal(nn.Module):
def __init__(self, n):
super().__init__()
self.capa = nn.Linear(n, 1)
def forward(self, x):
return self.capa(x).squeeze(-1)
class MLP(nn.Module):
def __init__(self, n, oculta=32, p=0.1):
super().__init__()
self.red = nn.Sequential(
nn.Linear(n, oculta), nn.ReLU(), nn.Dropout(p),
nn.Linear(oculta, oculta), nn.ReLU(), nn.Dropout(p),
nn.Linear(oculta, 1))
def forward(self, x):
return self.red(x).squeeze(-1)
def evaluar(modelo, d):
modelo.eval() # dropout off
with torch.no_grad(): # no graph
return nn.functional.mse_loss(modelo(d.X), d.y).item()
def entrenar(modelo, epocas=40, lr=1e-3, wd=0.0, paciencia=8,
semilla=0, restaurar=True):
opt = torch.optim.Adam(modelo.parameters(), lr=lr,
weight_decay=wd)
dl = cargador(semilla)
hist, mejor, sin_mejora = [], (np.inf, None, -1), 0
for epoca in range(epocas):
modelo.train() # dropout on
suma = 0.0
for xb, yb in dl:
opt.zero_grad()
perdida = nn.functional.mse_loss(modelo(xb), yb)
perdida.backward()
opt.step()
suma += perdida.item() * len(yb)
val = evaluar(modelo, ds["val"])
hist.append((epoca, suma / len(ds["ent"]), val))
if val < mejor[0]:
mejor = (val, copy.deepcopy(modelo.state_dict()), epoca)
sin_mejora = 0
else:
sin_mejora += 1
if sin_mejora >= paciencia:
break
if restaurar:
modelo.load_state_dict(mejor[1])
return pd.DataFrame(hist, columns=["epoca", "ent", "val"]), mejor[2]
torch.manual_seed(0)
lineal = Lineal(4)
hist_lin, ep_lin = entrenar(lineal, epocas=30, lr=1e-2, paciencia=30)
torch.manual_seed(0)
mlp = MLP(4)
hist_mlp, ep_mlp = entrenar(mlp, lr=1e-3, wd=1e-4)
n_par = sum(p.numel() for p in mlp.parameters())
print(f"linear: best epoch {ep_lin}; MLP ({n_par} parameters): "
f"best epoch {ep_mlp} of {len(hist_mlp)}")
```
```output
linear: best epoch 5; MLP (1249 parameters): best epoch 8 of 17
```
The details that separate a clean loop from one that “almost works”:
- `modelo.train()` and `modelo.eval()` switch dropout on and off (and, if there were any, batch normalization).
Evaluating in training mode gives a noisy validation loss that is worse than the real one.
- `opt.zero_grad()` before each `backward()`, because of gradient accumulation.
- Early stopping keeps a **copy** of the best `state_dict` (`copy.deepcopy`: without it, the dictionary points to the
same tensors, which keep being updated) and restores it at the end.
- The stopping epoch is chosen with validation, so validation is no longer out of sample for that model: the result
is measured on the test set.
A linear model with squared loss has a closed-form solution (least squares). If the loop is right, Adam has to get
close to it. It is a cheap check of everything above: data, standardization, loop and optimizer.
```python
Xe = np.c_[np.ones(len(ds["ent"])), ds["ent"].X.double().numpy()]
b_mco = np.linalg.lstsq(Xe, ds["ent"].y.double().numpy(),
rcond=None)[0]
b_torch = np.r_[lineal.capa.bias.item(),
lineal.capa.weight.detach().double().numpy()[0]]
print(pd.DataFrame({"OLS": b_mco, "Adam": b_torch},
index=["const"] + FEATS).round(3).T.to_string())
# in HAR units: slope per unit of log RV
print("HAR (OLS):", (b_mco[1:4] / desv[["d", "w", "m"]]).round(3)
.to_dict())
```
```output
const d w m neg
OLS 0.000 0.120 0.159 0.138 -0.035
Adam -0.005 0.124 0.175 0.137 -0.028
HAR (OLS): {'d': 0.2, 'w': 0.342, 'm': 0.342}
```
Adam ends up close to the closed-form solution, not on top of it: the slopes differ in the second digit (0.175 versus
0.159 on the 5-day mean), because of the noise of 256-row batches and because early stopping keeps epoch 5. In the
test error the difference does not show (next section). The last line translates the least squares coefficients into
HAR units: 0.20 for today's RV and 0.34 for each mean. The generator uses 0.36, 0.34 and 0.22 on the logarithm of
**integrated** variance; one day's RV measures it with error, and that error attenuates the daily coefficient and
shifts weight to the means, which average it out. It is what is expected, not a bug in the loop.
## The result, against the benchmark, and determinism
The figure at the top shows the validation curves of the two models by epoch. The linear model settles in five epochs
on the least squares value, 0.197, and oscillates around it because of batch noise; the MLP goes down to 0.191 at its
best epoch, the 8th, and early stopping cuts at the 16th, after eight epochs without improvement. What matters is
still missing: the test set, 2018-2025, which has not been touched to choose anything.
```python
def predecir(modelo, d):
modelo.eval()
with torch.no_grad():
return modelo(d.X).double().numpy()
dp = ds["prueba"]
y = dp.y.double().numpy()
Xp = np.c_[np.ones(len(dp)), dp.X.double().numpy()]
pred = {"naive (today's RV)": (dp.X[:, 0].double().numpy()
* desv["d"] + media["d"]),
"HAR by OLS": Xp @ b_mco,
"linear in PyTorch": predecir(lineal, dp),
"MLP": predecir(mlp, dp)}
ecm = {k: np.mean((y - p) ** 2) for k, p in pred.items()}
print(pd.Series(ecm).round(3).to_string())
# difference of squared errors, mean per week
dif = (y - pred["HAR by OLS"]) ** 2 - (y - pred["MLP"]) ** 2
semana = pd.to_datetime(h.loc[h["tramo"] == "prueba", "fecha"]
).dt.to_period("W").to_numpy()
s = pd.Series(dif).groupby(semana).mean()
print(f"MLP versus HAR: {dif.mean() / ecm['HAR by OLS']:.1%} "
f"lower MSE; t = {s.mean() / s.std() * np.sqrt(len(s)):.1f} "
f"({len(s)} weeks)")
```
```output
naive (today's RV) 0.319
HAR by OLS 0.213
linear in PyTorch 0.213
MLP 0.208
MLP versus HAR: 2.5% lower MSE; t = 6.3 (418 weeks)
```
*Fig. — Test mean squared error (2018-2025) of the forecast of tomorrow's log RV across the six assets: naive, HAR by least squares, the same linear model trained in PyTorch and the MLP.* Four horizontal bars. Naive 0.319; HAR by least squares 0.213; linear in PyTorch 0.213; MLP (highlighted) 0.208, 2.5% below HAR.
What the table says, in order:
- **The naive forecast loses by far.** Today's RV alone has a squared error 50% larger than the HAR: the mean over
several days removes a good part of the measurement error of a single day.
- **The linear model trained with Adam and the least squares one give the same** on the test set. The loop does what
it should.
- **The MLP beats the HAR by 2.5%**, with a weekly difference of t = 6.3 over 418 weeks. It is a real nonlinearity in
these data: the measurement error of the log RV is asymmetric (a χ² with 78 degrees of freedom in logarithms) and
the effect of negative returns is not linear in the return. But it is a small improvement over a four-coefficient
model; with real data it would have to be checked over several periods (lesson 5.4.3), discounting how many
architectures were tried (lesson 5.4.4), before believing it.
A **reproducible** result does not come for free. With what was fixed at the beginning of the lesson (seed,
deterministic algorithms, threads fixed by the environment) and the `DataLoader`'s dedicated generator, two training
runs with the same seed give the same weights, bit for bit; with another seed, different weights.
```python
def pesos(m):
return torch.cat([p.detach().flatten() for p in m.parameters()])
copias = []
for semilla in (0, 0, 1):
torch.manual_seed(semilla)
m = MLP(4)
entrenar(m, epocas=5, semilla=semilla)
copias.append(pesos(m))
print("seed 0 twice:", torch.equal(copias[0], copias[1]))
print("seed 0 versus 1:", torch.equal(copias[0], copias[2]))
```
```output
seed 0 twice: True
seed 0 versus 1: False
```
| What is fixed | How | What it prevents |
|---|---|---|
| Initialization and dropout | `torch.manual_seed(…)` before creating the model | different initial weights and masks |
| Batch order | `DataLoader(…, generator=torch.Generator().manual_seed(…))` | depending on the state of the global generator |
| Algorithms | `torch.use_deterministic_algorithms(True)` | operations whose results change between runs (above all on GPU) |
| Threads | fixed `OMP_NUM_THREADS` (the course runner sets 4) | floating-point sums split differently |
| Outputs | round to what the conclusion needs | far decimals that change between CPUs |
## GPU, MPS and torch.compile
On CPU, a 1,249-parameter network with 12,000 rows trains in seconds. Large networks need an accelerator. PyTorch
treats them as devices: the model and each batch are moved with `.to(device)` and everything else is the same.
```python
if torch.cuda.is_available():
dispositivo = torch.device("cuda")
elif torch.backends.mps.is_available():
dispositivo = torch.device("mps")
else:
dispositivo = torch.device("cpu")
print(dispositivo, torch.__version__)
```
```output
cpu 2.14.1+cpu
```
On a machine with an NVIDIA GPU, the training loop changes in three lines, and mixed precision is usually added
(`bfloat16` on recent GPUs). The code is not run in the course, which runs on CPU only:
```python skip
modelo = MLP(4).to(dispositivo)
opt = torch.optim.Adam(modelo.parameters(), lr=1e-3)
dl = DataLoader(ds["ent"], batch_size=4096, shuffle=True,
pin_memory=True,
generator=torch.Generator().manual_seed(0))
for xb, yb in dl:
xb = xb.to(dispositivo, non_blocking=True)
yb = yb.to(dispositivo, non_blocking=True)
opt.zero_grad()
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
perdida = nn.functional.mse_loss(modelo(xb), yb)
perdida.backward()
opt.step()
```
- **CUDA** (NVIDIA): the reference accelerator, and where determinism costs the most: many GPU operations sum in an
order that changes between runs. With `use_deterministic_algorithms(True)`, those without a deterministic version
raise a `RuntimeError` (PyTorch's reproducibility page gives the example of `index_add_`): better an error than a
silently different result.
- **MPS** (the GPU of Macs with Apple Silicon): same code with `"mps"`. Not all operations or types are implemented:
the missing ones fail or, with `PYTORCH_ENABLE_MPS_FALLBACK=1`, run on CPU. The `float64` checks in this course are
best done on CPU.
- **When it pays off**: with large models and batches. With a network like the one in this lesson, copying the
batches to the GPU costs more than the computation.
`torch.compile` (PyTorch 2) traces the model and generates code optimized for the device; on CPU, C++ code that is
built with the system compiler. The first call is slow (it compiles); the following ones, faster if the model is
large. With Python 3.14 and PyTorch 2.14 it works on CPU:
```python
import os
import tempfile
# the compilation cache, in a temporary folder
os.environ["TORCHINDUCTOR_CACHE_DIR"] = tempfile.mkdtemp()
compilado = torch.compile(mlp)
mlp.eval()
with torch.no_grad():
a_eager = mlp(dp.X)
a_comp = compilado(dp.X)
print("same predictions:", torch.allclose(a_eager, a_comp,
atol=1e-5))
```
```output
same predictions: True
```
Same predictions (to `float32` tolerance: the generated code may sum in a different order). Compilation took between
20 and 25 seconds on the machine the lesson was written on, for a network that predicts the 12,000 test days in
milliseconds: with small models it does not pay off. It makes sense with transformers and large networks, on GPU,
and when the same model is going to be run many times.
## The same network with Lightning
Lightning reorganizes PyTorch code: the model, the loss of each step and the optimizer go into a `LightningModule`,
and the loop is supplied by the `Trainer`, which adds what a large project ends up writing anyway: checkpoints, metric
logging, multiple GPUs, mixed precision, gradient accumulation. It is the base of `neuralforecast`, which is used in
lessons 5.7.2 and 5.7.3.
```python
import logging
import warnings
import lightning as L
for n in ("pytorch_lightning", "lightning.pytorch",
"lightning_fabric", "lightning"):
logging.getLogger(n).setLevel(logging.ERROR)
warnings.filterwarnings("ignore", module="pytorch_lightning")
warnings.filterwarnings("ignore", module="lightning")
class ModuloVol(L.LightningModule):
def __init__(self, red, lr=1e-3, wd=1e-4):
super().__init__()
self.red, self.lr, self.wd = red, lr, wd
def training_step(self, lote, i):
x, y = lote
return nn.functional.mse_loss(self.red(x), y)
def validation_step(self, lote, i):
x, y = lote
self.log("val", nn.functional.mse_loss(self.red(x), y))
def configure_optimizers(self):
return torch.optim.Adam(self.parameters(), lr=self.lr,
weight_decay=self.wd)
def con_lightning(epocas, gen_val):
torch.manual_seed(0)
red = MLP(4)
val = DataLoader(ds["val"], batch_size=len(ds["val"]),
generator=gen_val)
L.Trainer(max_epochs=epocas, accelerator="cpu",
deterministic=True, logger=False,
enable_progress_bar=False, enable_model_summary=False,
enable_checkpointing=False, num_sanity_val_steps=0,
).fit(ModuloVol(red), cargador(0), val)
return red
torch.manual_seed(0)
a_mano = MLP(4)
entrenar(a_mano, epocas=5, wd=1e-4, paciencia=99, restaurar=False)
for nombre, gen in [("without generator", None),
("with generator", torch.Generator())]:
red = con_lightning(5, gen)
print(f"Lightning, val {nombre}: weights identical to the "
f"handwritten loop: {torch.equal(pesos(red), pesos(a_mano))}")
```
```output
Lightning, val without generator: weights identical to the handwritten loop: False
Lightning, val with generator: weights identical to the handwritten loop: True
```
With the validation `DataLoader` **without** its own generator, the Lightning weights and those of the handwritten
loop agree in the first epoch and diverge afterward. The cause is not Lightning: each time a `DataLoader` is
iterated, its iterator draws a base seed from the loader's generator or, if it has none, from torch's **global
generator**. The handwritten loop evaluates without a `DataLoader`; Lightning iterates the validation one at the end
of each epoch, consumes a number from the global generator and changes the dropout masks of the next epoch. With a
dedicated generator in the validation loader, the weights are identical bit for bit. The lesson applies to any
project: everything random with its own generator, and nothing that depends on the order in which the rest of the
program consumes random numbers.
What Lightning does not do for you: split the data by date, purge, normalize with training statistics or choose the
metric. That stays in the `DataLoader`s and in your code, as in this lesson.
## Exercises
1. Check with finite differences the gradient of the MLP's validation loss with respect to three weights of its
first layer. Work in `float64` (`copy.deepcopy(mlp).double()`) and in `eval` mode (dropout would make the loss
random).
**Solution.**
```python
m64 = copy.deepcopy(mlp).double().eval()
Xv, yv = ds["val"].X.double(), ds["val"].y.double()
m64.zero_grad()
nn.functional.mse_loss(m64(Xv), yv).backward()
W = m64.red[0].weight
paso = 1e-5
for (i, j) in [(0, 0), (5, 2), (17, 3)]:
with torch.no_grad():
W[i, j] += paso
Lmas = nn.functional.mse_loss(m64(Xv), yv).item()
W[i, j] -= 2 * paso
Lmen = nn.functional.mse_loss(m64(Xv), yv).item()
W[i, j] += paso
fd = (Lmas - Lmen) / (2 * paso)
print(f"w[{i},{j}]: autograd {W.grad[i, j].item():+.8f} "
f"finite diff. {fd:+.8f}")
```
```output
w[0,0]: autograd +0.00071116 finite diff. +0.00071116
w[5,2]: autograd -0.00029703 finite diff. -0.00029703
w[17,3]: autograd +0.00020451 finite diff. +0.00020451
```
They agree in all eight printed digits. Two details: the weights are modified inside `torch.no_grad()` (otherwise,
autograd would record the modification in the graph), and they are restored to their value at the end. Functions
with “corners”, such as ReLU, give bad finite differences if the step crosses the zero of some neuron; with a step of
10⁻⁵ that is rare.
2. Remove `opt.zero_grad()` from the loop and train the linear model for 10 epochs with the same `lr`. What happens to
the validation loss and why?
**Solution.**
```python
def entrenar_sin_cero(modelo, epocas, lr):
opt = torch.optim.Adam(modelo.parameters(), lr=lr)
vals = []
for epoca in range(epocas):
modelo.train()
for xb, yb in cargador(0):
nn.functional.mse_loss(modelo(xb), yb).backward()
opt.step()
vals.append(round(evaluar(modelo, ds["val"]), 3))
return vals
torch.manual_seed(0)
print("without zero_grad:", entrenar_sin_cero(Lineal(4), 10, 1e-2))
print("with zero_grad: ", hist_lin["val"].round(3).tolist()[:10])
```
```output
without zero_grad: [0.432, 0.836, 2.191, 1.423, 2.192, 4.034, 4.052, 2.941, 9.944, 2.517]
with zero_grad: [0.221, 0.206, 0.2, 0.198, 0.198, 0.197, 0.199, 0.197, 0.198, 0.198]
```
Without `zero_grad`, the gradient of each step is the sum of all the previous ones in the epoch: a kind of undamped
momentum that pushes the weights away from the minimum, and the validation loss stays far above that of the correct
loop. The model raises no error: it just trains badly. It is the most common mistake when writing the loop by hand,
and a reason to encapsulate it (a function of your own, like `entrenar`, or Lightning).
3. A colleague builds the `Dataset` with all the rows, does `random_split(ds, [0.7, 0.15, 0.15])` and normalizes the
features with the mean of the whole sample. Their MLP gives a somewhat lower test error than the one in this
lesson. What would you tell them?
**Solution.**
That their test does not measure what they think. With `random_split`, each test day has its immediate neighbors in
training, with almost identical features (the 22-day mean changes little from one day to the next) and with targets
that are features of the adjacent row: one row's tomorrow RV is the next row's today RV. It is the leak of the random
k-fold of lesson 5.4.1. And normalizing with the whole sample uses the mean and standard deviation of 2018-2025 to
standardize 2008-2015: information from the future, small but systematic (lesson 5.2.4). Moreover, without date
splits there is no way to know how the model would have behaved through the level shifts of 2019 and 2022. The fix
is the one in this lesson: splits by label date, purging at the boundaries and training statistics; and, to compare
with theirs, the same procedure with both architectures.
## Self-assessment
1. You create `t = torch.from_numpy(a)` and then do `t[0, 0] = 9`. What happens to `a`?
- A. It changes too: `from_numpy` shares memory with the array; `torch.tensor(a)` would make a copy
- B. Nothing: `from_numpy` copies the data
- C. It raises an error because tensors are immutable
- D. It changes only if `a` is `float32`
*Answer A: It changes too: `from_numpy` shares memory with the array; `torch.tensor(a)` would make a copy.* `torch.from_numpy` and `Tensor.numpy()` share memory (on CPU). It is useful to avoid duplicating large data and dangerous if you modify one thinking it is a copy.
2. You compare autograd with central finite differences in `float32`, and the error is smallest with a step of 0.01 and grows with smaller steps. Why?
- A. Because autograd is inexact in `float32`
- B. Because central differences only work with large steps
- C. Because with small steps the subtraction of two nearly equal losses loses significant digits (rounding error ≈ ε/h), and with large steps truncation error dominates (≈ h²): the optimum lies between the two
- D. Because the function is not differentiable
*Answer C: Because with small steps the subtraction of two nearly equal losses loses significant digits (rounding error ≈ ε/h), and with large steps truncation error dominates (≈ h²): the optimum lies between the two.* With ε ≈ 1.2·10⁻⁷ in `float32`, the best step for central differences is around ε^(1/3) ≈ 0.005; in `float64` (ε ≈ 2.2·10⁻¹⁶), around 10⁻⁵. That is why gradient checks are done in `float64`.
3. The last training row is dated 2015-12-31 and its target is the RV of January 4, 2016, which is already in the validation set. What do you do?
- A. Nothing: it is a single row per asset
- B. Move it to validation
- C. Shuffle it with the rest
- D. Purge it: no training row may have a target that falls inside the validation period, just as in the purged k-fold of 5.4.2
*Answer D: Purge it: no training row may have a target that falls inside the validation period, just as in the purged k-fold of 5.4.2.* With a one-day horizon the leak is small, but the rule is the same as with 21-session labels: the split is defined by the date of the label, not by that of the features.
4. You train the same network twice with the same seed, once with your loop and once with Lightning, and the weights differ from the second epoch on. The batch order is the same. Most likely cause?
- A. Lightning uses another optimizer
- B. The validation `DataLoader` without its own `generator` draws its seed from torch's global generator when iterated, and that changes the dropout masks of the next epoch
- C. Lightning is not deterministic
- D. The CPU threads
*Answer B: The validation `DataLoader` without its own `generator` draws its seed from torch's global generator when iterated, and that changes the dropout masks of the next epoch.* Each time a `DataLoader` is iterated, its iterator takes a base seed from `loader.generator` or, if it has none, from the global generator. With its own `torch.Generator()` in the validation loader, the two training runs give identical weights.
5. The MLP lowers the test squared error of the linear HAR by 2.5%, with a weekly difference of t ≈ 6. Which conclusion is correct?
- A. That networks are always better than linear volatility models
- B. That there is a leak in the test
- C. That there is a small, measurable nonlinearity (the measurement error of log RV is not normal and the effect of negative returns is not linear), but the bulk of the forecast comes from the HAR structure, and the network is validated against it
- D. That the HAR is badly estimated
*Answer C: That there is a small, measurable nonlinearity (the measurement error of log RV is not normal and the effect of negative returns is not linear), but the bulk of the forecast comes from the HAR structure, and the network is validated against it.* The truth in the data is a HAR on the logarithm of integrated variance; RV adds an asymmetric measurement error. A network can scrape something there, never turn it into a different forecast. The linear benchmark is what sets the scale.
## Key points
- A tensor is an array with a `dtype`, a device and, if needed, a history for autograd. `from_numpy` shares memory; mixing NumPy's `float64` with `float32` layers raises an error.
- Autograd gets checked: against the analytical derivative when there is one and against central finite differences in `float64` (step ≈ 10⁻⁵). In `float32` the best step is around 10⁻² and the error does not go below ≈ 5·10⁻⁵.
- Splits are made by the label date, with purging at the boundaries and normalization with training statistics; shuffling within the training set is correct.
- The clean loop: `train()`/`eval()`, `zero_grad`, `backward`, `step`, evaluation under `no_grad` and early stopping that keeps the best `state_dict`. A linear model trained this way reaches the least squares solution.
- Determinism: seed, `use_deterministic_algorithms(True)`, fixed threads and a dedicated `torch.Generator` in each `DataLoader`. With that, the handwritten loop and Lightning give identical weights.
- `torch.compile` works on CPU with Python 3.14, but compiling costs seconds that a small network never recovers; the GPU (CUDA or MPS) pays off with large models and batches.
## Resources
### Documentation
- [PyTorch: tensors (torch.Tensor documentation)](https://docs.pytorch.org/docs/stable/tensors.html) · docs.pytorch.org
- [PyTorch: autograd mechanics](https://docs.pytorch.org/docs/stable/notes/autograd.html) · docs.pytorch.org
- [PyTorch: torch.utils.data (Dataset, DataLoader and randomness in data loading)](https://docs.pytorch.org/docs/stable/data.html) · docs.pytorch.org
- [PyTorch: reproducibility](https://docs.pytorch.org/docs/stable/notes/randomness.html) · docs.pytorch.org
- [PyTorch: introduction to torch.compile](https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) · docs.pytorch.org
- [Lightning: LightningModule and Trainer](https://lightning.ai/docs/pytorch/stable/common/lightning_module.html) · lightning.ai
### Data
- [Level 5 data: returns and realized variance of six assets (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/vol5_diario.parquet) · Download · vol5_diario.parquet
Previous lesson: [5.6.4 · Callbacks, a Sharpe loss and the model in production](https://pythonparatrading.com/en/python-trading-course/level-5/keras-callbacks-custom-loss-export/)
Next lesson: [5.7.2 · Time series architectures: TCN, N-BEATS, PatchTST and iTransformer](https://pythonparatrading.com/en/python-trading-course/level-5/tcn-n-beats-patchtst-itransformer/)