---
title: "5.9.2 RL for execution, market making and hedging"
description: "RL agents vs Almgren-Chriss, Avellaneda-Stoikov and the Black-Scholes delta, measured on the same fixed test paths."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/rl-execution-market-making-hedging/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/rl-execution-market-making-hedging.png"
---
# 5.9.2 · RL for execution, market making and dynamic hedging
> Where reinforcement learning has serious arguments in trading is in control problems with a credible simulator and a clear objective: liquidating a position, quoting as a market maker, hedging an option. This lesson trains an agent on each one and measures it on fixed test paths against the analytical solution that is already known. It wins where the reference makes assumptions the simulator does not satisfy; where it does not, at best it matches it.
- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.9 · Reinforcement learning
- Duration: 75 min
- Prerequisites: [5.9.1 · MDP, Q-learning, PPO and Gymnasium environments](https://pythonparatrading.com/en/python-trading-course/level-5/mdp-q-learning-ppo-gymnasium/), [4.3.1 · Implementation shortfall, VWAP, TWAP and Almgren-Chriss](https://pythonparatrading.com/en/python-trading-course/level-4/implementation-shortfall-vwap-twap-almgren-chriss/), [2.3.6 · Introduction to market making: inventory, spread and adverse selection (Avellaneda-Stoikov)](https://pythonparatrading.com/en/python-trading-course/level-2/market-making-avellaneda-stoikov/), [3.6.2 · Delta hedging, gamma scalping and dispersion](https://pythonparatrading.com/en/python-trading-course/level-3/delta-hedging-gamma-scalping-dispersion/)
## By the end of this lesson
- Design the reward of an execution problem so that its sum is the Almgren-Chriss mean-variance objective, and remove the zero-mean noise from it without changing the solution.
- Train PPO to liquidate a position and evaluate it on the 1,000 test paths against TWAP and Almgren-Chriss, with the error of the difference.
- Set up Avellaneda-Stoikov market making as an environment, compare the agent with the reference and with a direct parameter search, and understand which criterion is being compared.
- Program a lightweight deep hedging with PyTorch (differentiating through the simulation) and measure whether it improves on the Black-Scholes delta with costs under Heston.
- Tell, in each case, whether the agent contributes something the analytical model did not have.
## Where RL has arguments
Lesson 5.9.1 ended with a warning: an agent can only be judged if there is something to compare it with. In
directional trading there is not (and lesson 5.9.3 shows what happens). In three control problems, there is:
| Problem | Action | Objective | Analytical reference | Lesson |
|---|---|---|---|---|
| Optimal execution | how much to sell each minute | cost + λ·variance | Almgren-Chriss | 4.3.1 |
| Market making | where to place the quotes | P&L − inventory risk | Avellaneda-Stoikov | 2.3.6 |
| Dynamic hedging | how much of the underlying to hold | P&L − λ·variance | Black-Scholes delta | 3.6.2 |
All three share what directional trading lacks: a **simulator** that can be believed in the essentials (impact, order
arrivals, dynamics of the underlying), an **objective** that does not depend on predicting the market, and a
**reference solution** with known assumptions. The agent can beat the reference where the simulator does not satisfy
those assumptions; where it does, the best it can do is match it. The lesson's method is always the same:
```mermaid
flowchart LR
A[Simulator] --> B[Train agent]
B --> C[Test paths]
D[Reference] --> C
C --> E[Difference and error]
```
The test paths are fixed and shared: the same for the agent and for the references, never seen during training. They
are in the level 5 data, with the parameters and reference figures the README already publishes.
## Execution: the reward that adds up to the Almgren-Chriss objective
The problem is the one in lesson 4.3.1 with linear impact: sell X = 200,000 shares at S0 = 50 USD in 60 one-minute
steps. The mid follows $S_k = S_{k-1} + \sigma\sqrt{\tau}\,\xi_k - \tau\gamma v_k$ (permanent impact $\gamma$) and the
sale in step $k$, $n_k$ shares at rate $v_k = n_k/\tau$, executes at $S_{k-1} - \varepsilon - \eta v_k$ (spread and
temporary impact). The cost is the *implementation shortfall* $C = X S_0 - \sum_k n_k \cdot \text{price}_k$ and the
objective is to minimize $\mathbb{E}[C] + \lambda \operatorname{Var}[C]$. Almgren and Chriss give the optimal
inventory trajectory in closed form, $x_k = X \sinh(\kappa(T - t_k)) / \sinh(\kappa T)$, which is in the JSON.
```python
import json
import math
import numpy as np
import pandas as pd
cfg = json.load(open("rl5_entornos.json", encoding="utf-8"))
ej = cfg["ejecucion"]
X, N, S0, TAU = ej["X"], ej["N"], ej["S0"], ej["tau"]
SIG, ETA, GAM = ej["sigma"], ej["eta"], ej["gamma"]
EPS, LAM = ej["epsilon"], ej["lam"]
x_ac = np.array(ej["trayectoria_optima"])
x_tw = X * (1 - np.arange(N + 1) / N)
cam = pd.read_parquet("rl5_caminos_ejecucion.parquet")
xi = (cam.pivot(index="camino", columns="paso", values="xi")
.to_numpy()[:, 1:]) # step 0 is S0, with no innovation
print(xi.shape, f"kappa*T = {ej['kappa'] * N:.2f}",
f"daily sigma = {SIG * math.sqrt(390) / S0:.1%}")
```
```output
(1000, 60) kappa*T = 2.01 daily sigma = 2.0%
```
Before training anything, the simulator has to reproduce the published figures. Two functions: the realized cost on
some paths and the theoretical objective of a deterministic trajectory (Almgren-Chriss formulas):
```python
def coste_is(x, xi):
"""Implementation shortfall (USD) of selling according to the
inventory x (N+1 values) on the innovation paths xi (m, N)."""
n = -np.diff(x)
v = n / TAU
dS = SIG * math.sqrt(TAU) * xi - TAU * GAM * v
S = S0 + np.cumsum(dS, axis=1)
S_ant = np.column_stack([np.full(len(xi), S0), S[:, :-1]])
return X * S0 - (n * (S_ant - EPS - ETA * v)).sum(axis=1)
def objetivo_teorico(x):
n = -np.diff(x)
E = (0.5 * GAM * X**2 + EPS * X
+ (ETA - 0.5 * GAM * TAU) / TAU * (n**2).sum())
V = SIG**2 * TAU * (x[1:] ** 2).sum()
return E + LAM * V
def resumen(c):
return {"media": c.mean(), "desv.": c.std(ddof=1),
"media + λ·var": c.mean() + LAM * c.var(ddof=1)}
ref = {"TWAP": x_tw, "Almgren-Chriss": x_ac}
tabla = pd.DataFrame({k: resumen(coste_is(x, xi))
for k, x in ref.items()}).T
tabla["teórico E + λV"] = [objetivo_teorico(x) for x in ref.values()]
tabla.round(0)
```
```output
media desv. media + λ·var teórico E + λV
TWAP 31866.0 46054.0 38228.0 36584.0
Almgren-Chriss 32301.0 36915.0 36389.0 35385.0
```
The figures match those in the README (38,228 and 36,389 USD measured; 36,584 and 35,385 theoretical): the simulator
is the same. Almgren-Chriss sells faster at the start (it pays more impact) to shed the price risk sooner, and beats
TWAP by 1,839 USD of objective on the test paths.
For an agent to optimize that objective, the reward of each step has to add up exactly to $-(C + \lambda V)$. Writing
the price of each sale as $S_0$ plus the previous moves, the cost is split across steps:
$$
C = \sum_{k=1}^{N} \Big[ n_k(\varepsilon + \eta v_k) + x_k \tau \gamma v_k - x_k \sigma \sqrt{\tau}\, \xi_k \Big]
$$
with $x_k$ the inventory remaining after sale $k$. The first term is the cost of executing; the second, the permanent
impact absorbed by the remaining inventory; the third, what is gained or lost from the price move on that inventory.
The variance of a deterministic trajectory is $\sigma^2\tau\sum_k x_k^2$, so the reward of step $k$ is
$$
r_k = -\Big[ n_k(\varepsilon + \eta v_k) + x_k \tau \gamma v_k + \lambda \sigma^2 \tau x_k^2 \Big] + x_k \sigma \sqrt{\tau}\, \xi_k
$$
The last term is zero-mean noise: $\xi_k$ is independent of the decision. It has a huge standard deviation (with
100,000 shares in the position, about 5,000 USD per minute) against differences of a few hundred USD between
strategies over the whole episode. Removing it does not change the expectation of the reward or the optimal policy,
and it greatly reduces its variance: it is a **control variate**, and the penalty $\lambda\sigma^2\tau x_k^2$ already
represents the risk. The environment supports both versions.
```python
import gymnasium as gym
from gymnasium import spaces
from gymnasium.utils.env_checker import check_env
MULT = np.array([0.5, 0.75, 1.0, 1.5, 2.0, 2.5])
class EjecucionEnv(gym.Env):
"""Observes (fraction of time, fraction of inventory); sells
MULT[a] times the TWAP rate of the remaining inventory."""
def __init__(self, ruido=True):
self.ruido = ruido
self.observation_space = spaces.Box(0.0, 1.0, (2,),
np.float32)
self.action_space = spaces.Discrete(len(MULT))
def _obs(self):
return np.array([self.k / N, self.x / X], np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.k, self.x = 0, float(X)
return self._obs(), {}
def step(self, a):
quedan = N - self.k
n = self.x if quedan == 1 else min(
self.x, self.x / quedan * MULT[int(a)])
v = n / TAU
self.x -= n
self.k += 1
coste = (n * (EPS + ETA * v) + self.x * TAU * GAM * v
+ LAM * SIG**2 * TAU * self.x**2)
if self.ruido:
xi_k = self.np_random.standard_normal()
coste -= self.x * SIG * math.sqrt(TAU) * xi_k
return self._obs(), -coste / 1000, self.k == N, False, {}
check_env(EjecucionEnv(), skip_render_check=True)
print("check_env: no errors")
```
```output
check_env: no errors
```
Design decisions. The action is a multiple of the TWAP rate of the remaining inventory (0.5 to 2.5 times), so any
policy sells everything on time and the last step liquidates whatever is left; Almgren-Chriss starts by selling about
2.1 times the TWAP rate and ends at 1, within the range. The observation does not include the price: in this model the
optimal trajectory does not depend on it, and giving it to the agent would only give it noise to react to. The reward
is in thousands of USD (a scale of order 1, as in 5.9.1). The episode truly ends when everything is sold
(`terminated`), and without discounting ($\gamma = 1$): the cost of a minute weighs the same at the start as at the
end.
## Execution: PPO with and without the noise
First, with the realized reward, the one an agent that only observes the P&L would see. PPO with 8 environments, no
discounting, `gae_lambda=1` (the advantage with the full return of the episode) and a learning rate of 0.001; after
each block of 30,000 steps (500 episodes) the agent's deterministic trajectory is read and its theoretical objective
computed:
```python
import torch
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
def trayectoria(modelo):
env = EjecucionEnv(ruido=False)
o, _ = env.reset(seed=0)
x = [float(X)]
for _ in range(N):
a, _ = modelo.predict(o, deterministic=True)
o, *_ = env.step(a)
x.append(env.x)
return np.array(x)
def entrena_ejecucion(ruido, bloques):
vec = make_vec_env(lambda: EjecucionEnv(ruido), n_envs=8,
seed=0)
m = PPO("MlpPolicy", vec, gamma=1.0, gae_lambda=1.0,
n_steps=240, batch_size=240, learning_rate=1e-3,
policy_kwargs={"net_arch": [32, 32]},
seed=0, device="cpu", verbose=0)
for b in range(bloques):
m.learn(30_000, reset_num_timesteps=b == 0)
x = trayectoria(m)
print(f"{30_000 * (b + 1):>6} steps: theoretical objective "
f"{objetivo_teorico(x):>7,.0f} USD; inventory at "
f"minute 20: {x[20]:>7,.0f}")
return m, x
print(f"references: AC {objetivo_teorico(x_ac):,.0f}, TWAP "
f"{objetivo_teorico(x_tw):,.0f}; AC inventory at "
f"minute 20: {x_ac[20]:,.0f}")
ppo_ruido, x_ruido = entrena_ejecucion(ruido=True, bloques=2)
```
```output
references: AC 35,385, TWAP 36,584; AC inventory at minute 20: 97,150
30000 steps: theoretical objective 35,623 USD; inventory at minute 20: 108,522
60000 steps: theoretical objective 36,275 USD; inventory at minute 20: 112,630
```
With the realized reward, the agent learns something in the first 30,000 steps (35,623 USD of theoretical objective,
better than TWAP and 238 from Almgren-Chriss) and loses it in the next 30,000 (36,275): its inventory at minute 20 rises
from 108,522 to 112,630 shares, further from the 97,150 of the optimal trajectory. The noise of each episode (tens of
thousands of USD) masks the differences between trajectories (a few hundred), and each PPO update pushes in the
direction of the episodes that got lucky. The result depends on where training is stopped, and with a real market
there would be no way of knowing which of the two to keep.
Now with the reward without the noise term, same expectation:
```python
ppo_exec, x_rl = entrena_ejecucion(ruido=False, bloques=3)
```
```output
30000 steps: theoretical objective 35,993 USD; inventory at minute 20: 71,420
60000 steps: theoretical objective 35,772 USD; inventory at minute 20: 82,671
90000 steps: theoretical objective 35,503 USD; inventory at minute 20: 94,107
```
Without the noise term, learning is stable: 35,993, 35,772 and 35,503 USD, each block closer to Almgren-Chriss's
35,385, and the inventory at minute 20 converges toward that of the optimal trajectory (94,107 vs 97,150). The header
figure draws the inventory trajectories: the agent reproduces the Almgren-Chriss curve, which sells fast at the start,
and moves away from the TWAP straight line. It is no coincidence that the improvement comes from the design of the
reward and not from the algorithm: in applied RL, the reward is half the model.
*Fig. — Almgren-Chriss objective (mean cost + λ·variance) on the 1,000 test paths: the two benchmarks and the two PPO agents.* Horizontal bars in thousand USD, axis starting at 34: TWAP 38,228, PPO with realized reward 37,646, PPO with expected reward 36,425 (highlighted) and Almgren-Chriss 36,389.
The evaluation that counts is on the 1,000 test paths. Since the three strategies are evaluated on the same paths, the
path-by-path cost difference has much less noise than each cost separately; its standard error comes from those
differences:
```python
estr = {"TWAP": x_tw, "Almgren-Chriss": x_ac,
"PPO, recompensa realizada": x_ruido,
"PPO, recompensa esperada": x_rl}
costes = {k: coste_is(x, xi) for k, x in estr.items()}
tabla = pd.DataFrame({k: resumen(c) for k, c in costes.items()}).T
d = costes["PPO, recompensa esperada"] - costes["Almgren-Chriss"]
print(f"PPO (expected) − AC, path by path: {d.mean():.0f} USD "
f"of mean cost (standard error {d.std(ddof=1) / 1000**0.5:.0f})")
tabla.round(0)
```
```output
PPO (expected) − AC, path by path: 332 USD of mean cost (standard error 63)
media desv. media + λ·var
TWAP 31866.0 46054.0 38228.0
Almgren-Chriss 32301.0 36915.0 36389.0
PPO, recompensa realizada 32206.0 42584.0 37646.0
PPO, recompensa esperada 32633.0 35551.0 36425.0
```
On the test paths, the agent with the expected reward ends up at 36,425 USD of mean + λ·variance: 36 above
Almgren-Chriss (36,389) and 1,803 below TWAP (38,228). Path by path, it pays 332 USD more in mean cost (standard error
63), because it sells somewhat faster at the start, and makes up for almost all of it with lower variance (standard
deviation of 35,551 vs 36,915). The realized-reward agent, stopped at 60,000 steps, ends up halfway: 37,646.
It is the expected result, and the good one. The simulator satisfies the Almgren-Chriss assumptions (linear impact,
arithmetic price, mean-variance objective), so its formula is optimal and an agent can only get close. It needed
90,000 steps (1,500 simulated episodes) and a redesigned reward to get where the formula gets with one line. Had it
ended up below Almgren-Chriss by more than its sampling error, the right thing would be to look for the mistake in the
environment, not to celebrate.
## Market making: the agent against Avellaneda-Stoikov
The market maker of lesson 2.3.6 quotes on both sides around a reservation price $r = s - q\gamma\sigma^2(T - t)$
that moves away from the mid against its inventory $q$, with a total spread
$\gamma\sigma^2(T - t) + \frac{2}{\gamma}\ln(1 + \gamma/k)$. Other participants' orders arrive with intensity
$A e^{-k\delta}$, which falls with the distance $\delta$ from the quote to the mid. Avellaneda and Stoikov derive those
formulas for an agent with exponential utility of final wealth (risk aversion $\gamma$), with approximations.
The README simulator (the paper's parameters: σ = 2, T = 1, dt = 0.005, γ = 0.1, A = 140, k = 1.5) is rewritten in
vectorized form, to evaluate any skew rule over 1,000 simulations with the same random numbers:
```python
mm = cfg["market_making"]
SM, SIGM, TM, DT = mm["S0"], mm["sigma"], mm["T"], mm["dt"]
GMM, KMM, AMM = mm["gamma"], mm["k"], mm["A"]
PASOS = int(round(TM / DT))
def horquilla(tl):
return GMM * SIGM**2 * tl + 2 / GMM * math.log(1 + GMM / KMM)
def simula_mm(sesgo, n=1000, semilla=592):
"""Final P&L and final |inventory| of quoting centered on
S + sesgo(q, time left) with the Avellaneda-Stoikov spread."""
rng = np.random.default_rng(semilla)
z = rng.choice([-1.0, 1.0], (n, PASOS))
u1, u2 = rng.random((n, PASOS)), rng.random((n, PASOS))
S, q, caja = np.full(n, SM), np.zeros(n), np.zeros(n)
for i in range(PASOS):
tl = TM - i * DT
centro, h = S + sesgo(q, tl), horquilla(tl) / 2
pa, pb = centro + h, centro - h
vende = u1[:, i] < AMM * np.exp(-KMM * (pa - S)) * DT
compra = u2[:, i] < AMM * np.exp(-KMM * (S - pb)) * DT
q += compra.astype(float) - vende
caja += vende * pa - compra * pb
S = S + SIGM * math.sqrt(DT) * z[:, i]
return caja + q * S, np.abs(q)
def resumen_mm(sesgo, **kw):
w, q = simula_mm(sesgo, **kw)
return {"P&L medio": w.mean(), "desv.": w.std(ddof=1),
"|q| final": q.mean(),
"media − γ/2·var": w.mean() - GMM / 2 * w.var(ddof=1)}
def avellaneda(q, tl):
return -q * GMM * SIGM**2 * tl
refs_mm = {"Avellaneda-Stoikov": avellaneda,
"simétrica": lambda q, tl: 0 * q}
pd.DataFrame({k: resumen_mm(f) for k, f in refs_mm.items()}).T.round(2)
```
```output
P&L medio desv. |q| final media − γ/2·var
Avellaneda-Stoikov 65.01 6.65 2.19 62.80
simétrica 67.68 13.89 6.97 58.03
```
With random numbers different from the README's, the figures change a little (65.01 of mean P&L for
Avellaneda-Stoikov, vs 65.12), but the pattern is the paper's: the inventory strategy earns somewhat less on average
than the symmetric one (65.01 vs 67.68) with half the standard deviation (6.65 vs 13.89) and a third of the final
inventory. On the criterion mean − (γ/2)·variance, the second-order approximation of exponential utility with
γ = 0.1, it wins clearly: 62.80 vs 58.03.
The environment gives the agent the same Avellaneda-Stoikov spread and lets it choose only the skew: the center of the
quotes is $s - \beta q$, with $\beta$ among six values (Avellaneda-Stoikov is equivalent to $\beta = 0.4(T - t)$, from
0.4 at the start to 0 at the end). The reward of each step is what is captured when a quote fills (the distance from
the quote to the mid) minus the inventory penalty $\frac{\gamma}{2}\sigma^2 q^2\,dt$, whose expected sum is the part of
the P&L variance due to inventory. The term $q\,\Delta S$ (gain or loss on the inventory from the move of the mid) has
zero mean and is removed, as in execution.
```python
BETAS = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.6])
class MMEnv(gym.Env):
def __init__(self):
self.observation_space = spaces.Box(-1.0, 1.0, (2,),
np.float32)
self.action_space = spaces.Discrete(len(BETAS))
def _obs(self):
return np.array([np.clip(self.q / 10, -1, 1), self.i / PASOS],
np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.S, self.q, self.i = SM, 0, 0
return self._obs(), {}
def step(self, a):
tl = TM - self.i * DT
centro = self.S - BETAS[int(a)] * self.q
h = horquilla(tl) / 2
pa, pb = centro + h, centro - h
u = self.np_random.random(3)
captura = 0.0
if u[0] < AMM * math.exp(-KMM * (pa - self.S)) * DT:
self.q -= 1
captura += pa - self.S
if u[1] < AMM * math.exp(-KMM * (self.S - pb)) * DT:
self.q += 1
captura += self.S - pb
self.S += SIGM * math.sqrt(DT) * (1.0 if u[2] < 0.5 else -1.0)
self.i += 1
r = captura - GMM / 2 * SIGM**2 * self.q**2 * DT
return self._obs(), r, self.i == PASOS, False, {}
check_env(MMEnv(), skip_render_check=True)
vec = make_vec_env(MMEnv, n_envs=8, seed=0)
ppo_mm = PPO("MlpPolicy", vec, gamma=0.99, n_steps=400,
batch_size=400, learning_rate=3e-4,
policy_kwargs={"net_arch": [32, 32]},
seed=0, device="cpu", verbose=0)
ppo_mm.learn(80_000)
def sesgo_agente(q, tl):
obs = np.column_stack([np.clip(q / 10, -1, 1),
np.full(len(q), (TM - tl) / TM)])
a, _ = ppo_mm.predict(obs.astype(np.float32), deterministic=True)
return -BETAS[a] * q
rejilla = [(q, t) for t in (0.0, 0.5, 0.9) for q in (-6, -2, 2, 6)]
obs = np.array([[q / 10, t] for q, t in rejilla], np.float32)
print("chosen β:", BETAS[ppo_mm.predict(obs, deterministic=True)[0]])
```
```output
chosen β: [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]
```
The agent, after 80,000 steps (400 simulated sessions), has learned a simpler rule than the paper's: β = 0.1 at all
twelve grid points, with any inventory and at any moment of the session. It does not reduce the skew toward the close,
as Avellaneda-Stoikov does, and it starts with a skew four times smaller.
```python
# direct search: constant β, chosen with other simulations
busqueda = {b: resumen_mm(lambda q, tl, b=b: -b * q, semilla=1592)
for b in BETAS}
mejor = max(busqueda, key=lambda b: busqueda[b]["media − γ/2·var"])
print("β chosen by direct search (seed 1592):", mejor)
todas = dict(refs_mm)
todas["agente PPO"] = sesgo_agente
todas[f"búsqueda directa, β = {mejor}"] = (
lambda q, tl: -mejor * q)
pd.DataFrame({k: resumen_mm(f) for k, f in todas.items()}).T.round(2)
```
```output
β chosen by direct search (seed 1592): 0.1
P&L medio desv. |q| final media − γ/2·var
Avellaneda-Stoikov 65.01 6.65 2.19 62.80
simétrica 67.68 13.89 6.97 58.03
agente PPO 66.58 6.99 1.16 64.13
búsqueda directa, β = 0.1 66.58 6.99 1.16 64.13
```
*Fig. — Market making on 1,000 common simulations of the Avellaneda-Stoikov model: mean P&L against its standard deviation for the paper's formula, symmetric quoting and the constant skew chosen by PPO and by direct search; in brackets, mean − (γ/2)·variance.* Three points. Avellaneda-Stoikov: standard deviation 6.65 and mean 65.01 (mean − γ/2·variance 62.80). Symmetric: 13.89 and 67.68 (58.03). PPO agent and identical direct search with β = 0.1 (highlighted): 6.99 and 66.58 (64.13).
The agent beats Avellaneda-Stoikov on the table's criterion: 64.13 vs 62.80, with a higher mean P&L (66.58 vs
65.01), a similar standard deviation (6.99 vs 6.65) and half the final inventory (1.16 vs 2.19). The figure places the
four strategies on the mean and standard deviation plane.
Has RL discovered something the paper did not know? No. Avellaneda-Stoikov is optimal for its objective (exponential
utility of final wealth, with an approximation of arrivals for small inventories and long sessions), not for the
mean − variance of this simulator with its time step. The agent optimizes a different criterion and wins on that
criterion. And the **direct search** (trying the six constant values of β on another 1,000 simulations and keeping
the best) finds exactly the same, β = 0.1, in a second and without networks. When the optimal policy fits in one
parameter, an RL agent is the most expensive way to find it. What RL could contribute lies in what this simulator
lacks: arrivals that depend on informed flow and adverse selection (lesson 2.3.6), queues in the book, several
correlated assets. There is no closed-form formula there, and that is also where the simulator is hardest to believe.
## Hedging: lightweight deep hedging under Heston
The last problem is the one in lesson 3.6.2: you sell an ATM call with 21 sessions to expiry on 100 shares, collecting
the Black-Scholes premium with σ = 20% (230.30 USD), and hedge at each session's close with the underlying, paying
0.05% of the notional traded. The underlying does not follow Black-Scholes but **Heston**: stochastic volatility with
negative correlation with the price (ρ = −0.7) and a volatility of volatility of 0.5. The Black-Scholes delta is the
optimal hedge without costs and with constant volatility; here neither holds.
*Deep hedging* (Buehler et al., 2019) frames hedging as a policy: a network decides each day's position from what is
observable, and it is trained by directly maximizing the risk criterion of the final P&L over many simulated paths.
Since the P&L is a differentiable function of the positions, the gradient is computed by backpropagation through the
whole simulation, without estimating advantages as PPO does: it is a direct policy search with a differentiable
simulator, the efficient relative of RL when the simulator allows it. The lesson's lightweight version:
- Network **inputs**, observable at each day's close: log(S/K), the time remaining, the current position and the
Black-Scholes delta with σ = 20%. The Heston variance is not observed.
- **Output**: a correction to the delta, $\text{position} = \text{clip}(\Delta_{BS} + 0.5\tanh(f_\theta), 0, 1)$. The
network starts from the delta and learns how far to deviate.
- **Criterion**: maximize mean − 0.01·variance of the final P&L in USD, the last column of the README table.
- **Training**: 20,000 Heston paths simulated in the lesson with another seed (same parameters as the JSON) and
minibatches of 4,000. **Evaluation**: the 2,000 test paths in the Parquet file, which the network does not see.
```python
co = cfg["cobertura"]
K, ACC, C_TR = co["S0"], co["acciones"], co["coste"]
ND, SIG_BS, PRIMA = co["T_dias"], co["sigma_impl"], co["prima_usd"]
cob = pd.read_parquet("rl5_caminos_cobertura.parquet")
S_prueba = torch.tensor(cob.pivot(index="camino", columns="dia",
values="precio").to_numpy())
def heston(m, rng):
"""Daily Heston paths (Euler with truncation)."""
dt = 1 / co["dias_ano"] / co["subpasos"]
S, v = np.full(m, co["S0"]), np.full(m, co["v0"])
sal = [S.copy()]
rho = co["rho"]
for _ in range(ND):
for _ in range(co["subpasos"]):
z1 = rng.standard_normal(m)
z2 = (rho * z1
+ math.sqrt(1 - rho**2) * rng.standard_normal(m))
vp = np.maximum(v, 0)
S = S * np.exp(-vp / 2 * dt + np.sqrt(vp * dt) * z1)
v = (v + co["kappa"] * (co["theta"] - vp) * dt
+ co["xi"] * np.sqrt(vp * dt) * z2)
sal.append(S.copy())
return torch.tensor(np.array(sal).T)
S_ent = heston(20_000, np.random.default_rng(11))
def delta_bs(S, tau):
d1 = ((torch.log(S / K) + SIG_BS**2 * tau / 2)
/ (SIG_BS * math.sqrt(tau)))
return 0.5 * (1 + torch.erf(d1 / math.sqrt(2)))
def pnl_cobertura(S, politica):
"""Final P&L (USD) of selling the call and hedging each close with
politica(S_d, tau, pos, delta) -> new position per share."""
m = S.shape[0]
pos = torch.zeros(m, dtype=S.dtype)
caja = torch.full((m,), PRIMA, dtype=S.dtype)
for d in range(ND):
tau = (ND - d) / co["dias_ano"]
nueva = politica(S[:, d], tau, pos, delta_bs(S[:, d], tau))
dq = (nueva - pos) * ACC
caja = caja - dq * S[:, d] - C_TR * torch.abs(dq) * S[:, d]
pos = nueva
final = pos * ACC * S[:, -1]
caja = caja + final - C_TR * torch.abs(final)
return caja - torch.clamp(S[:, -1] - K, min=0) * ACC
def criterio(p):
return float(p.mean() - 0.01 * p.var())
def banda(b):
return lambda S, tau, pos, de: torch.where(
torch.abs(de - pos) > b, de, pos)
refs_co = {"sin cubrir": lambda S, tau, pos, de: 0 * de,
"delta BS diaria": lambda S, tau, pos, de: de,
"delta BS con banda 0,05": banda(0.05)}
filas = {}
for k, f in refs_co.items():
p = pnl_cobertura(S_prueba, f)
filas[k] = {"media": float(p.mean()), "desv.": float(p.std()),
"cuantil 5 %": float(torch.quantile(p, 0.05)),
"media − 0,01·var": criterio(p)}
pd.DataFrame(filas).T.round(2)
```
```output
media desv. cuantil 5 % media − 0,01·var
sin cubrir -1.91 296.71 -604.05 -882.28
delta BS diaria -5.37 57.11 -98.08 -37.99
delta BS con banda 0,05 -4.64 58.05 -97.94 -38.34
```
The figures are the README's: the daily delta with costs gives a criterion of −37.99; the 0.05 band, which trades
less, does not improve it (−38.34). Unhedged, the criterion is −882: hedging reduces the standard deviation from 297
to 57 USD.
```python
torch.manual_seed(0)
red = torch.nn.Sequential(
torch.nn.Linear(4, 32), torch.nn.ReLU(),
torch.nn.Linear(32, 32), torch.nn.ReLU(),
torch.nn.Linear(32, 1)).double()
def politica_red(S, tau, pos, de):
entrada = torch.stack([torch.log(S / K) * 5,
torch.full_like(S, tau * 12), pos, de], 1)
return torch.clamp(de + 0.5 * torch.tanh(red(entrada)[:, 0]),
0, 1)
delta_ent = pnl_cobertura(S_ent, refs_co["delta BS diaria"])
print(f"BS delta on the training paths: "
f"{criterio(delta_ent):.2f}")
opt = torch.optim.Adam(red.parameters(), lr=1e-3)
gen = torch.Generator().manual_seed(0)
for it in range(1, 201):
lote = torch.randperm(len(S_ent), generator=gen)[:4000]
p = pnl_cobertura(S_ent[lote], politica_red)
perdida = -(p.mean() - 0.01 * p.var())
opt.zero_grad()
perdida.backward()
opt.step()
if it % 50 == 0:
with torch.no_grad():
ent = criterio(pnl_cobertura(S_ent, politica_red))
pru = criterio(pnl_cobertura(S_prueba, politica_red))
print(f"iteration {it:>3}: training criterion {ent:7.2f}"
f", test {pru:7.2f}")
```
```output
BS delta on the training paths: -42.26
iteration 50: training criterion -40.30, test -36.63
iteration 100: training criterion -39.79, test -36.12
iteration 150: training criterion -39.57, test -35.94
iteration 200: training criterion -39.45, test -35.84
```
Both criteria improve at the same time. On the training paths, from −42.26 with the delta to −39.45 with the network
(2.81 USD); on the test paths, from −37.99 to −35.84. The levels are not comparable between sets (they are different
paths and the training ones come out more dispersed), but the improvements are, and the test improvement is somewhat
smaller than the training one, as it should be. With 20,000 paths for about 1,200 parameters, overfitting to the
sample is small; with 2,000 paths it would be another story. In production it would be trained longer (here, 200 Adam
steps so the lesson runs in seconds) and with more paths.
```python
with torch.no_grad():
p_red = pnl_cobertura(S_prueba, politica_red)
p_bs = pnl_cobertura(S_prueba, refs_co["delta BS diaria"])
# transaction costs paid by each strategy
sin_coste = {}
for nombre, f in (("red", politica_red),
("delta", refs_co["delta BS diaria"])):
C_TR, guarda = 0.0, C_TR
sin_coste[nombre] = pnl_cobertura(S_prueba, f)
C_TR = guarda
for nombre, p in (("delta BS diaria", p_bs), ("deep hedging", p_red)):
clave = "delta" if nombre.startswith("delta") else "red"
costes = float((sin_coste[clave] - p).mean())
print(f"{nombre:<16} mean {float(p.mean()):6.2f} std "
f"{float(p.std()):5.2f} costs {costes:5.2f} "
f"criterion {criterio(p):7.2f}")
rng = np.random.default_rng(0)
mejoras = []
for _ in range(1000):
i = torch.from_numpy(rng.integers(0, len(p_red), len(p_red)))
mejoras.append(criterio(p_red[i]) - criterio(p_bs[i]))
print(f"criterion improvement: {criterio(p_red) - criterio(p_bs):.2f}"
f" USD (bootstrap standard error {np.std(mejoras):.2f})")
```
```output
delta BS diaria mean -5.37 std 57.11 costs 12.23 criterion -37.99
deep hedging mean -4.66 std 55.84 costs 11.23 criterion -35.84
criterion improvement: 2.14 USD (bootstrap standard error 0.65)
```
*Fig. — Mean − 0.01·variance of the P&L from hedging a short call under Heston on the 2,000 test paths: deep hedging against the Black-Scholes delta with and without bands (less negative is better).* Negative horizontal bars: BS delta without costs −25.30 (no-cost bound, from the README), deep hedging −35.84 (highlighted), daily BS delta −37.99, delta with 0.05 band −38.34 and with 0.10 band −43.24 (README).
The network improves the delta's criterion by 2.14 USD, with a bootstrap standard error of 0.65 (more than three
standard errors), and it does so through the two routes theory anticipates: it trades somewhat less (11.23 USD of
transaction costs vs 12.23, 8% less) and hedges somewhat better under Heston (standard deviation of 55.84 vs 57.11).
The figure compares the criterion of the hedges. It is a small improvement, about 2 USD on a premium of 230, but a
systematic one: in an options book hedged every day, it is money. And it comes where the fixed no-trade band, the
manual rule for trading less, made the result worse (−38.34).
## What the agent contributes and what it does not
| Problem | Reference | Agent on the test paths | Why |
|---|---|---|---|
| Execution | Almgren-Chriss | +36 USD of objective: equal | the simulator satisfies its assumptions |
| Market making | Avellaneda-Stoikov | 64.13 vs 62.80, same as the direct search | a different criterion; one parameter is enough |
| Hedging | BS delta with costs | +2.14 USD of criterion (standard error 0.65) | Heston and costs break its assumptions |
The pattern across the three cases is the one the introduction anticipated. Where the simulator satisfies the
assumptions of the analytical solution (execution), the agent matches it, at a higher cost. Where the agent seems to
beat the reference (market making), it is worth asking on which criterion: if the business wants the mean − variance
of the session's P&L, the useful question is why use a formula derived for a different objective, and a direct search
answers it more cheaply than an agent. Where the reference makes assumptions the simulator does not satisfy (hedging
with costs under Heston), a learned model improves, by a little and measurably. In all three, what makes the result
credible is the same: a known reference, shared test paths the agent has not seen and the standard error of the
difference.
## Exercises
1. Repeat the direct search for the market-making skew with a more risk-averse criterion, mean − 0.2·variance
(instead of γ/2 = 0.05). Which β does it choose and how does Avellaneda-Stoikov fare on that criterion?
**Solution.**
```python
def crit_02(sesgo, semilla):
w, _ = simula_mm(sesgo, semilla=semilla)
return w.mean() - 0.2 * w.var(ddof=1)
b02 = {b: crit_02(lambda q, tl, b=b: -b * q, 1592) for b in BETAS}
mejor02 = max(b02, key=b02.get)
print("chosen β:", mejor02)
prueba = crit_02(lambda q, tl: -mejor02 * q, 592)
print(f"on the test: search {prueba:.2f}, "
f"Avellaneda-Stoikov {crit_02(avellaneda, 592):.2f}")
```
```output
chosen β: 0.1
on the test: search 56.80, Avellaneda-Stoikov 56.17
```
With a criterion four times more risk-averse, the search again chooses β = 0.1 (the grid is coarse: 0.1 and 0.2 are
the only reasonable candidates) and stays ahead of Avellaneda-Stoikov on the test (56.80 vs 56.17), but the gap shrinks
from 1.33 to 0.63. The more the variance weighs, the closer the formula gets, since it reduces inventory hard at the
start of the session. With a finer grid or with a time-dependent β the result could change; it is exactly the kind of
question a direct search answers in seconds.
2. Evaluate the already trained deep hedging network on Heston paths with a volatility of volatility of 0.8 instead of
0.5 (2,000 new paths, seed 12), alongside the BS delta. Does the improvement hold outside the training simulator?
**Solution.**
```python
co_orig = co["xi"]
co["xi"] = 0.8
S_otro = heston(2_000, np.random.default_rng(12))
co["xi"] = co_orig
with torch.no_grad():
for nombre, f in (("delta BS diaria", refs_co["delta BS diaria"]),
("deep hedging", politica_red)):
p = pnl_cobertura(S_otro, f)
print(f"{nombre:<16} std {float(p.std()):6.2f} "
f"criterion {criterio(p):8.2f}")
```
```output
delta BS diaria std 77.14 criterion -59.02
deep hedging std 73.55 criterion -52.53
```
With ξ = 0.8 both hedges get worse (standard deviation of 77.14 and 73.55 USD) and the network's advantage grows:
6.49 USD of criterion (−52.53 vs −59.02). The correction it learned, trading somewhat less and adjusting the hedge to
the negative correlation between price and volatility, also points in the right direction with more volatility of
volatility. It is not a guarantee: changing a parameter within the same model family measures robustness to
calibration, not to model error (jumps, regimes, costs that are not proportional).
3. In the execution environment, what would happen if the price move since the start were added to the observation and
the agent were trained with the realized reward? Could the agent improve on Almgren-Chriss?
**Solution.**
In this model, no. The price follows an arithmetic random walk with independent innovations and the impact does not
depend on the price, so the past move says nothing about the future: the optimal trajectory for mean − λ·variance is
deterministic (Almgren and Chriss call it static, and for this criterion and this model strategies that react to the
price only improve marginally and with a different formulation of the objective). With the realized reward, the agent
would see spurious correlations between the observed price and the reward of each episode and would learn to react to
them: on the training paths it might look better; on the test paths, it would be equal to or worse than
Almgren-Chriss, with more variance. It is the overfitting of lesson 5.9.3 in miniature. If the price had mean
reversion or the impact depended on the state of the book, the observation would contribute and an agent could beat
the static formula, which is exactly what work such as Nevmyvaka, Feng and Kearns (2006) explores with real order book
data.
## Self-assessment
1. In the execution environment, the reward of each step includes the term x_k·σ·√τ·ξ_k, the gain or loss from the price move on the remaining inventory. Why can it be removed during training without changing the optimal policy?
- A. Because it is small compared with the impact costs
- B. Because its expectation is zero given any decision (ξ_k is independent of x_k), and the risk it represents is already in the penalty λ·σ²·τ·x_k²; removing it reduces the variance of the reward, it does not change its expectation
- C. Because Almgren-Chriss assumes the price does not move
- D. Because PPO does not accept negative rewards
*Answer B: Because its expectation is zero given any decision (ξ_k is independent of x_k), and the risk it represents is already in the penalty λ·σ²·τ·x_k²; removing it reduces the variance of the reward, it does not change its expectation.* It is a control variate: the expected objective is the same with or without it. With it, the reward of an episode has a standard deviation of tens of thousands of USD against differences of a few hundred between strategies, and PPO cannot tell them apart well.
2. The execution agent trained with the noise-free reward ends up, on the test paths, 36 USD from Almgren-Chriss in mean + λ·variance, and 1,803 ahead of TWAP. How should this be read?
- A. That it has learned a trajectory almost identical to the optimal one: in this model (linear impact, arithmetic price), the analytical solution is exact, and the most an agent can do is match it
- B. That the agent has beaten Almgren-Chriss
- C. That TWAP is the best strategy
- D. That there is a leak in the environment
*Answer A: That it has learned a trajectory almost identical to the optimal one: in this model (linear impact, arithmetic price), the analytical solution is exact, and the most an agent can do is match it.* When the simulator satisfies the assumptions of the analytical model, RL can only reproduce it, at a higher cost. If the agent won by more than the sampling error, there would be a mistake in the environment (the data README warns about this).
3. In market making, a constant inventory skew (β = 0.1) gives a higher mean − (γ/2)·variance than the Avellaneda-Stoikov formula. What do you conclude?
- A. That Avellaneda-Stoikov is wrong
- B. That the agent is smarter than the model
- C. That the formula is optimal for its own objective (exponential utility with approximations) and the agent optimizes a different one (mean − variance of the P&L with this simulator): the comparison depends on the criterion, and a direct search over the parameter gives the same as PPO
- D. That market making has no inventory risk
*Answer C: That the formula is optimal for its own objective (exponential utility with approximations) and the agent optimizes a different one (mean − variance of the P&L with this simulator): the comparison depends on the criterion, and a direct search over the parameter gives the same as PPO.* Beating a reference on a criterion that is not its own is not a win for RL. What is useful is the question it forces you to ask: what objective the business has and whether the reference optimizes it.
4. The lesson's deep hedging improves on the mean − 0.01·variance of the P&L of the Black-Scholes delta with costs on the 2,000 test paths. Why is it plausible that a learned model wins here?
- A. Because the network sees the true Heston variance
- B. Because the test paths were used for training
- C. Because neural networks always beat formulas
- D. Because the Black-Scholes delta is optimal only without costs and with constant volatility; under Heston with negative correlation and with costs, trading a bit less and adjusting the hedge to the skew reduces variance and costs
*Answer D: Because the Black-Scholes delta is optimal only without costs and with constant volatility; under Heston with negative correlation and with costs, trading a bit less and adjusting the hedge to the skew reduces variance and costs.* The agent wins where the reference makes assumptions the simulator does not satisfy. It is trained on its own paths and evaluated on the test paths, which it has never seen; the improvement is reported with its standard error.
5. What is the main risk of taking any of these three agents to production?
- A. That PPO is not deterministic
- B. That there are few test paths
- C. That the agent learns the simulator: if real impact is not linear, or volatility does not follow Heston, what it learned may be worse than the formula, and on test paths from the same simulator that does not show
- D. That the networks are too small
*Answer C: That the agent learns the simulator: if real impact is not linear, or volatility does not follow Heston, what it learned may be worse than the formula, and on test paths from the same simulator that does not show.* The test paths measure overfitting to the training seed, not model error. You need to test the agent in simulators with different assumptions (the square-root impact of 4.3.1, other Heston parameters) and, with real data, with an execution backtest.
## Key points
- All three problems have a simulator, a clear objective (mean − λ·variance) and an analytical reference: that is where RL in trading makes sense and where it can be measured.
- Execution: with the realized reward, PPO learns and unlearns (35,623 and 36,275 USD of theoretical objective at 30,000 and 60,000 steps); removing the zero-mean noise, it ends up 36 USD from Almgren-Chriss on the test paths (36,425 vs 36,389; TWAP, 38,228).
- When the simulator satisfies the assumptions of the analytical solution, the agent at best matches it. If it wins, there is a mistake in the environment.
- Market making: the agent chooses a constant skew β = 0.1 that gives 64.13 of mean − (γ/2)·variance vs 62.80 for Avellaneda-Stoikov, because it optimizes a different criterion; a direct search over the parameter finds the same in a second.
- Hedging: a lightweight deep hedging (a 32 × 32 network, trained on 20,000 of its own Heston paths) improves the criterion of the BS delta with costs by 2.14 USD (standard error 0.65) on the 2,000 test paths, with lower standard deviation and lower costs. It wins because the BS delta is optimal under assumptions that Heston and the costs do not satisfy.
- The risk is the same in all three: the agent learns the simulator. The test paths measure overfitting to the seed, not model error.
## Resources
### Research
- [Quantitative Models and Algorithmic Execution](https://pythonparatrading.com/en/quantitative-models-algorithmic-execution/) · Research-0009 · 11 min
### Documentation
- [Almgren, R. and Chriss, N. (2000/2001): Optimal execution of portfolio transactions. Journal of Risk 3(2), 5-39](https://doi.org/10.21314/JOR.2001.041) · doi.org
- [Avellaneda, M. and Stoikov, S. (2008): High-frequency trading in a limit order book. Quantitative Finance 8(3)](https://doi.org/10.1080/14697680701381228) · doi.org
- [Buehler, H., Gonon, L., Teichmann, J. and Wood, B. (2019): Deep hedging. Quantitative Finance 19(8). arXiv:1802.03042](https://arxiv.org/abs/1802.03042) · arxiv.org
- [Nevmyvaka, Y., Feng, Y. and Kearns, M. (2006): Reinforcement learning for optimized trade execution. ICML](https://doi.org/10.1145/1143844.1143929) · doi.org
- [Stable-Baselines3: PPO](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html) · stable-baselines3.readthedocs.io
### Data
- [Level 5 data: parameters and references of the RL environments (JSON)](https://pythonparatrading.com/curso/datos/nivel-5/rl5_entornos.json) · Download · rl5_entornos.json
- [Level 5 data: Heston hedging test paths (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/rl5_caminos_cobertura.parquet) · Download · rl5_caminos_cobertura.parquet
Previous lesson: [5.9.1 · MDP, Q-learning, PPO and Gymnasium environments](https://pythonparatrading.com/en/python-trading-course/level-5/mdp-q-learning-ppo-gymnasium/)
Next lesson: [5.9.3 · Why reinforcement learning overfits in directional trading](https://pythonparatrading.com/en/python-trading-course/level-5/why-rl-overfits-directional-trading/)