RL for execution, market making and dynamic hedging.

Where RL has arguments

Lesson 5.9.1 ended with a warning: an agent can only be judged if there is something to compare it with. In directional trading there is not (and lesson 5.9.3 shows what happens). In three control problems, there is:

All three share what directional trading lacks: a simulator that can be believed in the essentials (impact, order arrivals, dynamics of the underlying), an objective that does not depend on predicting the market, and a reference solution with known assumptions. The agent can beat the reference where the simulator does not satisfy those assumptions; where it does, the best it can do is match it. The lesson's method is always the same:

flowchart LR
  A[Simulator] --> B[Train agent]
  B --> C[Test paths]
  D[Reference] --> C
  C --> E[Difference and error]

The test paths are fixed and shared: the same for the agent and for the references, never seen during training. They are in the level 5 data, with the parameters and reference figures the README already publishes.

Execution: the reward that adds up to the Almgren-Chriss objective

The problem is the one in lesson 4.3.1 with linear impact: sell X = 200,000 shares at S0 = 50 USD in 60 one-minute steps. The mid follows $S_k = S_{k-1} + \sigma\sqrt{\tau}\,\xi_k - \tau\gamma v_k$ (permanent impact $\gamma$) and the sale in step $k$, $n_k$ shares at rate $v_k = n_k/\tau$, executes at $S_{k-1} - \varepsilon - \eta v_k$ (spread and temporary impact). The cost is the implementation shortfall $C = X S_0 - \sum_k n_k \cdot \text{price}_k$ and the objective is to minimize $\mathbb{E}[C] + \lambda \operatorname{Var}[C]$. Almgren and Chriss give the optimal inventory trajectory in closed form, $x_k = X \sinh(\kappa(T - t_k)) / \sinh(\kappa T)$, which is in the JSON.

import json
import math
import numpy as np
import pandas as pd

cfg = json.load(open("rl5_entornos.json", encoding="utf-8"))
ej = cfg["ejecucion"]
X, N, S0, TAU = ej["X"], ej["N"], ej["S0"], ej["tau"]
SIG, ETA, GAM = ej["sigma"], ej["eta"], ej["gamma"]
EPS, LAM = ej["epsilon"], ej["lam"]
x_ac = np.array(ej["trayectoria_optima"])
x_tw = X * (1 - np.arange(N + 1) / N)

cam = pd.read_parquet("rl5_caminos_ejecucion.parquet")
xi = (cam.pivot(index="camino", columns="paso", values="xi")
      .to_numpy()[:, 1:])  # step 0 is S0, with no innovation
print(xi.shape, f"kappa*T = {ej['kappa'] * N:.2f}",
      f"daily sigma = {SIG * math.sqrt(390) / S0:.1%}")

(1000, 60) kappa*T = 2.01 daily sigma = 2.0%

Before training anything, the simulator has to reproduce the published figures. Two functions: the realized cost on some paths and the theoretical objective of a deterministic trajectory (Almgren-Chriss formulas):

def coste_is(x, xi):
    """Implementation shortfall (USD) of selling according to the
    inventory x (N+1 values) on the innovation paths xi (m, N)."""
    n = -np.diff(x)
    v = n / TAU
    dS = SIG * math.sqrt(TAU) * xi - TAU * GAM * v
    S = S0 + np.cumsum(dS, axis=1)
    S_ant = np.column_stack([np.full(len(xi), S0), S[:, :-1]])
    return X * S0 - (n * (S_ant - EPS - ETA * v)).sum(axis=1)

def objetivo_teorico(x):
    n = -np.diff(x)
    E = (0.5 * GAM * X**2 + EPS * X
         + (ETA - 0.5 * GAM * TAU) / TAU * (n**2).sum())
    V = SIG**2 * TAU * (x[1:] ** 2).sum()
    return E + LAM * V

def resumen(c):
    return {"media": c.mean(), "desv.": c.std(ddof=1),
            "media + λ·var": c.mean() + LAM * c.var(ddof=1)}

ref = {"TWAP": x_tw, "Almgren-Chriss": x_ac}
tabla = pd.DataFrame({k: resumen(coste_is(x, xi))
                      for k, x in ref.items()}).T
tabla["teórico E + λV"] = [objetivo_teorico(x) for x in ref.values()]
tabla.round(0)

                  media    desv.  media + λ·var  teórico E + λV
TWAP            31866.0  46054.0        38228.0         36584.0
Almgren-Chriss  32301.0  36915.0        36389.0         35385.0

The figures match those in the README (38,228 and 36,389 USD measured; 36,584 and 35,385 theoretical): the simulator is the same. Almgren-Chriss sells faster at the start (it pays more impact) to shed the price risk sooner, and beats TWAP by 1,839 USD of objective on the test paths.

For an agent to optimize that objective, the reward of each step has to add up exactly to $-(C + \lambda V)$. Writing the price of each sale as $S_0$ plus the previous moves, the cost is split across steps:

with $x_k$ the inventory remaining after sale $k$. The first term is the cost of executing; the second, the permanent impact absorbed by the remaining inventory; the third, what is gained or lost from the price move on that inventory. The variance of a deterministic trajectory is $\sigma^2\tau\sum_k x_k^2$, so the reward of step $k$ is

The last term is zero-mean noise: $\xi_k$ is independent of the decision. It has a huge standard deviation (with 100,000 shares in the position, about 5,000 USD per minute) against differences of a few hundred USD between strategies over the whole episode. Removing it does not change the expectation of the reward or the optimal policy, and it greatly reduces its variance: it is a control variate, and the penalty $\lambda\sigma^2\tau x_k^2$ already represents the risk. The environment supports both versions.

import gymnasium as gym
from gymnasium import spaces
from gymnasium.utils.env_checker import check_env

MULT = np.array([0.5, 0.75, 1.0, 1.5, 2.0, 2.5])

class EjecucionEnv(gym.Env):
    """Observes (fraction of time, fraction of inventory); sells
    MULT[a] times the TWAP rate of the remaining inventory."""

    def __init__(self, ruido=True):
        self.ruido = ruido
        self.observation_space = spaces.Box(0.0, 1.0, (2,),
                                            np.float32)
        self.action_space = spaces.Discrete(len(MULT))

    def _obs(self):
        return np.array([self.k / N, self.x / X], np.float32)

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.k, self.x = 0, float(X)
        return self._obs(), {}

    def step(self, a):
        quedan = N - self.k
        n = self.x if quedan == 1 else min(
            self.x, self.x / quedan * MULT[int(a)])
        v = n / TAU
        self.x -= n
        self.k += 1
        coste = (n * (EPS + ETA * v) + self.x * TAU * GAM * v
                 + LAM * SIG**2 * TAU * self.x**2)
        if self.ruido:
            xi_k = self.np_random.standard_normal()
            coste -= self.x * SIG * math.sqrt(TAU) * xi_k
        return self._obs(), -coste / 1000, self.k == N, False, {}

check_env(EjecucionEnv(), skip_render_check=True)
print("check_env: no errors")

check_env: no errors

Design decisions. The action is a multiple of the TWAP rate of the remaining inventory (0.5 to 2.5 times), so any policy sells everything on time and the last step liquidates whatever is left; Almgren-Chriss starts by selling about 2.1 times the TWAP rate and ends at 1, within the range. The observation does not include the price: in this model the optimal trajectory does not depend on it, and giving it to the agent would only give it noise to react to. The reward is in thousands of USD (a scale of order 1, as in 5.9.1). The episode truly ends when everything is sold (terminated), and without discounting ($\gamma = 1$): the cost of a minute weighs the same at the start as at the end.

Execution: PPO with and without the noise

First, with the realized reward, the one an agent that only observes the P&L would see. PPO with 8 environments, no discounting, gae_lambda=1 (the advantage with the full return of the episode) and a learning rate of 0.001; after each block of 30,000 steps (500 episodes) the agent's deterministic trajectory is read and its theoretical objective computed:

import torch
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)

def trayectoria(modelo):
    env = EjecucionEnv(ruido=False)
    o, _ = env.reset(seed=0)
    x = [float(X)]
    for _ in range(N):
        a, _ = modelo.predict(o, deterministic=True)
        o, *_ = env.step(a)
        x.append(env.x)
    return np.array(x)

def entrena_ejecucion(ruido, bloques):
    vec = make_vec_env(lambda: EjecucionEnv(ruido), n_envs=8,
                       seed=0)
    m = PPO("MlpPolicy", vec, gamma=1.0, gae_lambda=1.0,
            n_steps=240, batch_size=240, learning_rate=1e-3,
            policy_kwargs={"net_arch": [32, 32]},
            seed=0, device="cpu", verbose=0)
    for b in range(bloques):
        m.learn(30_000, reset_num_timesteps=b == 0)
        x = trayectoria(m)
        print(f"{30_000 * (b + 1):>6} steps: theoretical objective "
              f"{objetivo_teorico(x):>7,.0f} USD; inventory at "
              f"minute 20: {x[20]:>7,.0f}")
    return m, x

print(f"references: AC {objetivo_teorico(x_ac):,.0f}, TWAP "
      f"{objetivo_teorico(x_tw):,.0f}; AC inventory at "
      f"minute 20: {x_ac[20]:,.0f}")
ppo_ruido, x_ruido = entrena_ejecucion(ruido=True, bloques=2)

references: AC 35,385, TWAP 36,584; AC inventory at minute 20: 97,150
 30000 steps: theoretical objective  35,623 USD; inventory at minute 20: 108,522
 60000 steps: theoretical objective  36,275 USD; inventory at minute 20: 112,630

With the realized reward, the agent learns something in the first 30,000 steps (35,623 USD of theoretical objective, better than TWAP and 238 from Almgren-Chriss) and loses it in the next 30,000 (36,275): its inventory at minute 20 rises from 108,522 to 112,630 shares, further from the 97,150 of the optimal trajectory. The noise of each episode (tens of thousands of USD) masks the differences between trajectories (a few hundred), and each PPO update pushes in the direction of the episodes that got lucky. The result depends on where training is stopped, and with a real market there would be no way of knowing which of the two to keep.

Now with the reward without the noise term, same expectation:

ppo_exec, x_rl = entrena_ejecucion(ruido=False, bloques=3)

 30000 steps: theoretical objective  35,993 USD; inventory at minute 20:  71,420
 60000 steps: theoretical objective  35,772 USD; inventory at minute 20:  82,671
 90000 steps: theoretical objective  35,503 USD; inventory at minute 20:  94,107

Without the noise term, learning is stable: 35,993, 35,772 and 35,503 USD, each block closer to Almgren-Chriss's 35,385, and the inventory at minute 20 converges toward that of the optimal trajectory (94,107 vs 97,150). The header figure draws the inventory trajectories: the agent reproduces the Almgren-Chriss curve, which sells fast at the start, and moves away from the TWAP straight line. It is no coincidence that the improvement comes from the design of the reward and not from the algorithm: in applied RL, the reward is half the model.

The evaluation that counts is on the 1,000 test paths. Since the three strategies are evaluated on the same paths, the path-by-path cost difference has much less noise than each cost separately; its standard error comes from those differences:

estr = {"TWAP": x_tw, "Almgren-Chriss": x_ac,
        "PPO, recompensa realizada": x_ruido,
        "PPO, recompensa esperada": x_rl}
costes = {k: coste_is(x, xi) for k, x in estr.items()}
tabla = pd.DataFrame({k: resumen(c) for k, c in costes.items()}).T
d = costes["PPO, recompensa esperada"] - costes["Almgren-Chriss"]
print(f"PPO (expected) − AC, path by path: {d.mean():.0f} USD "
      f"of mean cost (standard error {d.std(ddof=1) / 1000**0.5:.0f})")
tabla.round(0)

PPO (expected) − AC, path by path: 332 USD of mean cost (standard error 63)
                             media    desv.  media + λ·var
TWAP                       31866.0  46054.0        38228.0
Almgren-Chriss             32301.0  36915.0        36389.0
PPO, recompensa realizada  32206.0  42584.0        37646.0
PPO, recompensa esperada   32633.0  35551.0        36425.0

On the test paths, the agent with the expected reward ends up at 36,425 USD of mean + λ·variance: 36 above Almgren-Chriss (36,389) and 1,803 below TWAP (38,228). Path by path, it pays 332 USD more in mean cost (standard error 63), because it sells somewhat faster at the start, and makes up for almost all of it with lower variance (standard deviation of 35,551 vs 36,915). The realized-reward agent, stopped at 60,000 steps, ends up halfway: 37,646.

It is the expected result, and the good one. The simulator satisfies the Almgren-Chriss assumptions (linear impact, arithmetic price, mean-variance objective), so its formula is optimal and an agent can only get close. It needed 90,000 steps (1,500 simulated episodes) and a redesigned reward to get where the formula gets with one line. Had it ended up below Almgren-Chriss by more than its sampling error, the right thing would be to look for the mistake in the environment, not to celebrate.

Market making: the agent against Avellaneda-Stoikov

The market maker of lesson 2.3.6 quotes on both sides around a reservation price $r = s - q\gamma\sigma^2(T - t)$ that moves away from the mid against its inventory $q$, with a total spread $\gamma\sigma^2(T - t) + \frac{2}{\gamma}\ln(1 + \gamma/k)$. Other participants' orders arrive with intensity $A e^{-k\delta}$, which falls with the distance $\delta$ from the quote to the mid. Avellaneda and Stoikov derive those formulas for an agent with exponential utility of final wealth (risk aversion $\gamma$), with approximations.

The README simulator (the paper's parameters: σ = 2, T = 1, dt = 0.005, γ = 0.1, A = 140, k = 1.5) is rewritten in vectorized form, to evaluate any skew rule over 1,000 simulations with the same random numbers:

mm = cfg["market_making"]
SM, SIGM, TM, DT = mm["S0"], mm["sigma"], mm["T"], mm["dt"]
GMM, KMM, AMM = mm["gamma"], mm["k"], mm["A"]
PASOS = int(round(TM / DT))

def horquilla(tl):
    return GMM * SIGM**2 * tl + 2 / GMM * math.log(1 + GMM / KMM)

def simula_mm(sesgo, n=1000, semilla=592):
    """Final P&L and final |inventory| of quoting centered on
    S + sesgo(q, time left) with the Avellaneda-Stoikov spread."""
    rng = np.random.default_rng(semilla)
    z = rng.choice([-1.0, 1.0], (n, PASOS))
    u1, u2 = rng.random((n, PASOS)), rng.random((n, PASOS))
    S, q, caja = np.full(n, SM), np.zeros(n), np.zeros(n)
    for i in range(PASOS):
        tl = TM - i * DT
        centro, h = S + sesgo(q, tl), horquilla(tl) / 2
        pa, pb = centro + h, centro - h
        vende = u1[:, i] < AMM * np.exp(-KMM * (pa - S)) * DT
        compra = u2[:, i] < AMM * np.exp(-KMM * (S - pb)) * DT
        q += compra.astype(float) - vende
        caja += vende * pa - compra * pb
        S = S + SIGM * math.sqrt(DT) * z[:, i]
    return caja + q * S, np.abs(q)

def resumen_mm(sesgo, **kw):
    w, q = simula_mm(sesgo, **kw)
    return {"P&L medio": w.mean(), "desv.": w.std(ddof=1),
            "|q| final": q.mean(),
            "media − γ/2·var": w.mean() - GMM / 2 * w.var(ddof=1)}

def avellaneda(q, tl):
    return -q * GMM * SIGM**2 * tl

refs_mm = {"Avellaneda-Stoikov": avellaneda,
           "simétrica": lambda q, tl: 0 * q}
pd.DataFrame({k: resumen_mm(f) for k, f in refs_mm.items()}).T.round(2)

                    P&L medio  desv.  |q| final  media − γ/2·var
Avellaneda-Stoikov      65.01   6.65       2.19            62.80
simétrica               67.68  13.89       6.97            58.03

With random numbers different from the README's, the figures change a little (65.01 of mean P&L for Avellaneda-Stoikov, vs 65.12), but the pattern is the paper's: the inventory strategy earns somewhat less on average than the symmetric one (65.01 vs 67.68) with half the standard deviation (6.65 vs 13.89) and a third of the final inventory. On the criterion mean − (γ/2)·variance, the second-order approximation of exponential utility with γ = 0.1, it wins clearly: 62.80 vs 58.03.

The environment gives the agent the same Avellaneda-Stoikov spread and lets it choose only the skew: the center of the quotes is $s - \beta q$, with $\beta$ among six values (Avellaneda-Stoikov is equivalent to $\beta = 0.4(T - t)$, from 0.4 at the start to 0 at the end). The reward of each step is what is captured when a quote fills (the distance from the quote to the mid) minus the inventory penalty $\frac{\gamma}{2}\sigma^2 q^2\,dt$, whose expected sum is the part of the P&L variance due to inventory. The term $q\,\Delta S$ (gain or loss on the inventory from the move of the mid) has zero mean and is removed, as in execution.

BETAS = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.6])

class MMEnv(gym.Env):
    def __init__(self):
        self.observation_space = spaces.Box(-1.0, 1.0, (2,),
                                            np.float32)
        self.action_space = spaces.Discrete(len(BETAS))

    def _obs(self):
        return np.array([np.clip(self.q / 10, -1, 1), self.i / PASOS],
                        np.float32)

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.S, self.q, self.i = SM, 0, 0
        return self._obs(), {}

    def step(self, a):
        tl = TM - self.i * DT
        centro = self.S - BETAS[int(a)] * self.q
        h = horquilla(tl) / 2
        pa, pb = centro + h, centro - h
        u = self.np_random.random(3)
        captura = 0.0
        if u[0] < AMM * math.exp(-KMM * (pa - self.S)) * DT:
            self.q -= 1
            captura += pa - self.S
        if u[1] < AMM * math.exp(-KMM * (self.S - pb)) * DT:
            self.q += 1
            captura += self.S - pb
        self.S += SIGM * math.sqrt(DT) * (1.0 if u[2] < 0.5 else -1.0)
        self.i += 1
        r = captura - GMM / 2 * SIGM**2 * self.q**2 * DT
        return self._obs(), r, self.i == PASOS, False, {}

check_env(MMEnv(), skip_render_check=True)
vec = make_vec_env(MMEnv, n_envs=8, seed=0)
ppo_mm = PPO("MlpPolicy", vec, gamma=0.99, n_steps=400,
             batch_size=400, learning_rate=3e-4,
             policy_kwargs={"net_arch": [32, 32]},
             seed=0, device="cpu", verbose=0)
ppo_mm.learn(80_000)

def sesgo_agente(q, tl):
    obs = np.column_stack([np.clip(q / 10, -1, 1),
                           np.full(len(q), (TM - tl) / TM)])
    a, _ = ppo_mm.predict(obs.astype(np.float32), deterministic=True)
    return -BETAS[a] * q

rejilla = [(q, t) for t in (0.0, 0.5, 0.9) for q in (-6, -2, 2, 6)]
obs = np.array([[q / 10, t] for q, t in rejilla], np.float32)
print("chosen β:", BETAS[ppo_mm.predict(obs, deterministic=True)[0]])

chosen β: [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]

The agent, after 80,000 steps (400 simulated sessions), has learned a simpler rule than the paper's: β = 0.1 at all twelve grid points, with any inventory and at any moment of the session. It does not reduce the skew toward the close, as Avellaneda-Stoikov does, and it starts with a skew four times smaller.

# direct search: constant β, chosen with other simulations
busqueda = {b: resumen_mm(lambda q, tl, b=b: -b * q, semilla=1592)
            for b in BETAS}
mejor = max(busqueda, key=lambda b: busqueda[b]["media − γ/2·var"])
print("β chosen by direct search (seed 1592):", mejor)
todas = dict(refs_mm)
todas["agente PPO"] = sesgo_agente
todas[f"búsqueda directa, β = {mejor}"] = (
    lambda q, tl: -mejor * q)
pd.DataFrame({k: resumen_mm(f) for k, f in todas.items()}).T.round(2)

β chosen by direct search (seed 1592): 0.1
                           P&L medio  desv.  |q| final  media − γ/2·var
Avellaneda-Stoikov             65.01   6.65       2.19            62.80
simétrica                      67.68  13.89       6.97            58.03
agente PPO                     66.58   6.99       1.16            64.13
búsqueda directa, β = 0.1      66.58   6.99       1.16            64.13

The agent beats Avellaneda-Stoikov on the table's criterion: 64.13 vs 62.80, with a higher mean P&L (66.58 vs 65.01), a similar standard deviation (6.99 vs 6.65) and half the final inventory (1.16 vs 2.19). The figure places the four strategies on the mean and standard deviation plane.

Has RL discovered something the paper did not know? No. Avellaneda-Stoikov is optimal for its objective (exponential utility of final wealth, with an approximation of arrivals for small inventories and long sessions), not for the mean − variance of this simulator with its time step. The agent optimizes a different criterion and wins on that criterion. And the direct search (trying the six constant values of β on another 1,000 simulations and keeping the best) finds exactly the same, β = 0.1, in a second and without networks. When the optimal policy fits in one parameter, an RL agent is the most expensive way to find it. What RL could contribute lies in what this simulator lacks: arrivals that depend on informed flow and adverse selection (lesson 2.3.6), queues in the book, several correlated assets. There is no closed-form formula there, and that is also where the simulator is hardest to believe.

Hedging: lightweight deep hedging under Heston

The last problem is the one in lesson 3.6.2: you sell an ATM call with 21 sessions to expiry on 100 shares, collecting the Black-Scholes premium with σ = 20% (230.30 USD), and hedge at each session's close with the underlying, paying 0.05% of the notional traded. The underlying does not follow Black-Scholes but Heston: stochastic volatility with negative correlation with the price (ρ = −0.7) and a volatility of volatility of 0.5. The Black-Scholes delta is the optimal hedge without costs and with constant volatility; here neither holds.

Deep hedging (Buehler et al., 2019) frames hedging as a policy: a network decides each day's position from what is observable, and it is trained by directly maximizing the risk criterion of the final P&L over many simulated paths. Since the P&L is a differentiable function of the positions, the gradient is computed by backpropagation through the whole simulation, without estimating advantages as PPO does: it is a direct policy search with a differentiable simulator, the efficient relative of RL when the simulator allows it. The lesson's lightweight version:

• Network inputs, observable at each day's close: log(S/K), the time remaining, the current position and the Black-Scholes delta with σ = 20%. The Heston variance is not observed.
• Output: a correction to the delta, $\text{position} = \text{clip}(\Delta_{BS} + 0.5\tanh(f_\theta), 0, 1)$. The network starts from the delta and learns how far to deviate.
• Criterion: maximize mean − 0.01·variance of the final P&L in USD, the last column of the README table.
• Training: 20,000 Heston paths simulated in the lesson with another seed (same parameters as the JSON) and minibatches of 4,000. Evaluation: the 2,000 test paths in the Parquet file, which the network does not see.

co = cfg["cobertura"]
K, ACC, C_TR = co["S0"], co["acciones"], co["coste"]
ND, SIG_BS, PRIMA = co["T_dias"], co["sigma_impl"], co["prima_usd"]

cob = pd.read_parquet("rl5_caminos_cobertura.parquet")
S_prueba = torch.tensor(cob.pivot(index="camino", columns="dia",
                                  values="precio").to_numpy())

def heston(m, rng):
    """Daily Heston paths (Euler with truncation)."""
    dt = 1 / co["dias_ano"] / co["subpasos"]
    S, v = np.full(m, co["S0"]), np.full(m, co["v0"])
    sal = [S.copy()]
    rho = co["rho"]
    for _ in range(ND):
        for _ in range(co["subpasos"]):
            z1 = rng.standard_normal(m)
            z2 = (rho * z1
                  + math.sqrt(1 - rho**2) * rng.standard_normal(m))
            vp = np.maximum(v, 0)
            S = S * np.exp(-vp / 2 * dt + np.sqrt(vp * dt) * z1)
            v = (v + co["kappa"] * (co["theta"] - vp) * dt
                 + co["xi"] * np.sqrt(vp * dt) * z2)
        sal.append(S.copy())
    return torch.tensor(np.array(sal).T)

S_ent = heston(20_000, np.random.default_rng(11))

def delta_bs(S, tau):
    d1 = ((torch.log(S / K) + SIG_BS**2 * tau / 2)
          / (SIG_BS * math.sqrt(tau)))
    return 0.5 * (1 + torch.erf(d1 / math.sqrt(2)))

def pnl_cobertura(S, politica):
    """Final P&L (USD) of selling the call and hedging each close with
    politica(S_d, tau, pos, delta) -> new position per share."""
    m = S.shape[0]
    pos = torch.zeros(m, dtype=S.dtype)
    caja = torch.full((m,), PRIMA, dtype=S.dtype)
    for d in range(ND):
        tau = (ND - d) / co["dias_ano"]
        nueva = politica(S[:, d], tau, pos, delta_bs(S[:, d], tau))
        dq = (nueva - pos) * ACC
        caja = caja - dq * S[:, d] - C_TR * torch.abs(dq) * S[:, d]
        pos = nueva
    final = pos * ACC * S[:, -1]
    caja = caja + final - C_TR * torch.abs(final)
    return caja - torch.clamp(S[:, -1] - K, min=0) * ACC

def criterio(p):
    return float(p.mean() - 0.01 * p.var())

def banda(b):
    return lambda S, tau, pos, de: torch.where(
        torch.abs(de - pos) > b, de, pos)

refs_co = {"sin cubrir": lambda S, tau, pos, de: 0 * de,
           "delta BS diaria": lambda S, tau, pos, de: de,
           "delta BS con banda 0,05": banda(0.05)}
filas = {}
for k, f in refs_co.items():
    p = pnl_cobertura(S_prueba, f)
    filas[k] = {"media": float(p.mean()), "desv.": float(p.std()),
                "cuantil 5 %": float(torch.quantile(p, 0.05)),
                "media − 0,01·var": criterio(p)}
pd.DataFrame(filas).T.round(2)

                         media   desv.  cuantil 5 %  media − 0,01·var
sin cubrir               -1.91  296.71      -604.05           -882.28
delta BS diaria          -5.37   57.11       -98.08            -37.99
delta BS con banda 0,05  -4.64   58.05       -97.94            -38.34

The figures are the README's: the daily delta with costs gives a criterion of −37.99; the 0.05 band, which trades less, does not improve it (−38.34). Unhedged, the criterion is −882: hedging reduces the standard deviation from 297 to 57 USD.

torch.manual_seed(0)
red = torch.nn.Sequential(
    torch.nn.Linear(4, 32), torch.nn.ReLU(),
    torch.nn.Linear(32, 32), torch.nn.ReLU(),
    torch.nn.Linear(32, 1)).double()

def politica_red(S, tau, pos, de):
    entrada = torch.stack([torch.log(S / K) * 5,
                           torch.full_like(S, tau * 12), pos, de], 1)
    return torch.clamp(de + 0.5 * torch.tanh(red(entrada)[:, 0]),
                       0, 1)

delta_ent = pnl_cobertura(S_ent, refs_co["delta BS diaria"])
print(f"BS delta on the training paths: "
      f"{criterio(delta_ent):.2f}")
opt = torch.optim.Adam(red.parameters(), lr=1e-3)
gen = torch.Generator().manual_seed(0)
for it in range(1, 201):
    lote = torch.randperm(len(S_ent), generator=gen)[:4000]
    p = pnl_cobertura(S_ent[lote], politica_red)
    perdida = -(p.mean() - 0.01 * p.var())
    opt.zero_grad()
    perdida.backward()
    opt.step()
    if it % 50 == 0:
        with torch.no_grad():
            ent = criterio(pnl_cobertura(S_ent, politica_red))
            pru = criterio(pnl_cobertura(S_prueba, politica_red))
        print(f"iteration {it:>3}: training criterion {ent:7.2f}"
              f", test {pru:7.2f}")

BS delta on the training paths: -42.26
iteration  50: training criterion  -40.30, test  -36.63
iteration 100: training criterion  -39.79, test  -36.12
iteration 150: training criterion  -39.57, test  -35.94
iteration 200: training criterion  -39.45, test  -35.84

Both criteria improve at the same time. On the training paths, from −42.26 with the delta to −39.45 with the network (2.81 USD); on the test paths, from −37.99 to −35.84. The levels are not comparable between sets (they are different paths and the training ones come out more dispersed), but the improvements are, and the test improvement is somewhat smaller than the training one, as it should be. With 20,000 paths for about 1,200 parameters, overfitting to the sample is small; with 2,000 paths it would be another story. In production it would be trained longer (here, 200 Adam steps so the lesson runs in seconds) and with more paths.

with torch.no_grad():
    p_red = pnl_cobertura(S_prueba, politica_red)
    p_bs = pnl_cobertura(S_prueba, refs_co["delta BS diaria"])
    # transaction costs paid by each strategy
    sin_coste = {}
    for nombre, f in (("red", politica_red),
                      ("delta", refs_co["delta BS diaria"])):
        C_TR, guarda = 0.0, C_TR
        sin_coste[nombre] = pnl_cobertura(S_prueba, f)
        C_TR = guarda
for nombre, p in (("delta BS diaria", p_bs), ("deep hedging", p_red)):
    clave = "delta" if nombre.startswith("delta") else "red"
    costes = float((sin_coste[clave] - p).mean())
    print(f"{nombre:<16} mean {float(p.mean()):6.2f}  std "
          f"{float(p.std()):5.2f}  costs {costes:5.2f}  "
          f"criterion {criterio(p):7.2f}")
rng = np.random.default_rng(0)
mejoras = []
for _ in range(1000):
    i = torch.from_numpy(rng.integers(0, len(p_red), len(p_red)))
    mejoras.append(criterio(p_red[i]) - criterio(p_bs[i]))
print(f"criterion improvement: {criterio(p_red) - criterio(p_bs):.2f}"
      f" USD (bootstrap standard error {np.std(mejoras):.2f})")

delta BS diaria  mean  -5.37  std 57.11  costs 12.23  criterion  -37.99
deep hedging     mean  -4.66  std 55.84  costs 11.23  criterion  -35.84
criterion improvement: 2.14 USD (bootstrap standard error 0.65)

The network improves the delta's criterion by 2.14 USD, with a bootstrap standard error of 0.65 (more than three standard errors), and it does so through the two routes theory anticipates: it trades somewhat less (11.23 USD of transaction costs vs 12.23, 8% less) and hedges somewhat better under Heston (standard deviation of 55.84 vs 57.11). The figure compares the criterion of the hedges. It is a small improvement, about 2 USD on a premium of 230, but a systematic one: in an options book hedged every day, it is money. And it comes where the fixed no-trade band, the manual rule for trading less, made the result worse (−38.34).

What the agent contributes and what it does not

The pattern across the three cases is the one the introduction anticipated. Where the simulator satisfies the assumptions of the analytical solution (execution), the agent matches it, at a higher cost. Where the agent seems to beat the reference (market making), it is worth asking on which criterion: if the business wants the mean − variance of the session's P&L, the useful question is why use a formula derived for a different objective, and a direct search answers it more cheaply than an agent. Where the reference makes assumptions the simulator does not satisfy (hedging with costs under Heston), a learned model improves, by a little and measurably. In all three, what makes the result credible is the same: a known reference, shared test paths the agent has not seen and the standard error of the difference.

Exercises

1. Repeat the direct search for the market-making skew with a more risk-averse criterion, mean − 0.2·variance (instead of γ/2 = 0.05). Which β does it choose and how does Avellaneda-Stoikov fare on that criterion?

def crit_02(sesgo, semilla):
    w, _ = simula_mm(sesgo, semilla=semilla)
    return w.mean() - 0.2 * w.var(ddof=1)

b02 = {b: crit_02(lambda q, tl, b=b: -b * q, 1592) for b in BETAS}
mejor02 = max(b02, key=b02.get)
print("chosen β:", mejor02)
prueba = crit_02(lambda q, tl: -mejor02 * q, 592)
print(f"on the test: search {prueba:.2f}, "
      f"Avellaneda-Stoikov {crit_02(avellaneda, 592):.2f}")

chosen β: 0.1
on the test: search 56.80, Avellaneda-Stoikov 56.17

2. Evaluate the already trained deep hedging network on Heston paths with a volatility of volatility of 0.8 instead of 0.5 (2,000 new paths, seed 12), alongside the BS delta. Does the improvement hold outside the training simulator?

co_orig = co["xi"]
co["xi"] = 0.8
S_otro = heston(2_000, np.random.default_rng(12))
co["xi"] = co_orig
with torch.no_grad():
    for nombre, f in (("delta BS diaria", refs_co["delta BS diaria"]),
                      ("deep hedging", politica_red)):
        p = pnl_cobertura(S_otro, f)
        print(f"{nombre:<16} std {float(p.std()):6.2f}  "
              f"criterion {criterio(p):8.2f}")

delta BS diaria  std  77.14  criterion   -59.02
deep hedging     std  73.55  criterion   -52.53

3. In the execution environment, what would happen if the price move since the start were added to the observation and the agent were trained with the realized reward? Could the agent improve on Almgren-Chriss?