---
title: "5.9.1 MDP, Q-learning, PPO and Gymnasium environments"
description: "A trading MDP with costs and a known optimal policy: value iteration, tabular Q-learning, a Gymnasium environment, and PPO and SAC vs the optimum."
url: "https://pythonparatrading.com/en/python-trading-course/level-5/mdp-q-learning-ppo-gymnasium/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/mdp-q-learning-ppo-gymnasium.png"
---

# 5.9.1 · MDP, Q-learning, PPO and Gymnasium environments

> Reinforcement learning promises to learn to trade by trial and error. Before believing it in a market you have to see it work where the answer is known: a small trading problem, with costs, whose optimal policy comes out exactly by value iteration. This lesson sets it up as an MDP, solves it, learns it with Q-learning, packages it as a Gymnasium environment and measures PPO and SAC against the optimum.

- Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/)
- Module: 5.9 · Reinforcement learning
- Duration: 70 min
- Prerequisites: [2.3.6 · Introduction to market making: inventory, spread and adverse selection (Avellaneda-Stoikov)](https://pythonparatrading.com/en/python-trading-course/level-2/market-making-avellaneda-stoikov/), [2.4.7 · Cost modeling: commissions, bid-ask spread, impact, financing and securities borrowing](https://pythonparatrading.com/en/python-trading-course/level-2/backtest-transaction-cost-modeling/), [5.1.1 · Why financial machine learning is different](https://pythonparatrading.com/en/python-trading-course/level-5/why-financial-machine-learning-is-different/)

## By the end of this lesson

- Set up a trading problem as a Markov decision process: states, actions, transitions, a reward with costs and discounting.
- Compute the exact optimal policy with value iteration and evaluate any tabular policy without simulating, using the stationary distribution.
- Program tabular Q-learning and tell convergence of the policy apart from convergence of the values.
- Write a `gymnasium.Env` environment with the costs inside the reward and validate it with `check_env`.
- Train Stable-Baselines3 PPO and SAC deterministically and compare them with the optimum on the same paths.

## From backtest to sequential decision

Everything level 5 has done so far is supervised learning: a table of features, a label and a model that predicts the
label. The trading decision came afterward, as a fixed rule on top of the prediction (buy the top decile, lesson
5.4.4). Reinforcement learning (RL) brings the two together: an **agent** observes the state, chooses an action, the
**environment** returns a reward and a new state, and the agent learns the decision rule that maximizes the sum of
future rewards. There is no label: only the consequence of each action.

```mermaid
flowchart LR
  A[Agent] -->|action| E[Environment]
  E -->|reward| A
  E -->|new state| A
```

That shape fits trading problems where today's action changes tomorrow's problem: an open position that costs money to
close, an inventory that has to be liquidated (lesson 4.3.1), the quotes of a market maker that accumulate inventory
(lesson 2.3.6). The price of the flexibility is high: the agent learns by trial and error, with noisy rewards, and
nothing guarantees that what it learns is optimal. That is why this lesson starts with a problem whose exact answer can
be computed, and only then lets the algorithms look for it.

<aside class="callout" data-kind="aviso">

The MDP in this lesson is a toy with a signal-to-noise ratio far higher than that of any market (a mean of 5 bp vs a
standard deviation of 20 bp per step): it is chosen so that the algorithms converge in seconds and what they learn can
be compared with the exact optimum. Lesson 5.9.3 does the same with a realistic signal, and the result is different.

</aside>

## A trading problem as an MDP

A **Markov decision process** (MDP) is a tuple $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$: states, actions, transition
probabilities $P(s' \mid s, a)$, expected reward $R(s, a)$ and discount factor $\gamma \in [0, 1)$. “Markov” means
that the current state summarizes everything about the past that matters for what comes next.

The lesson's problem:

- Market **regime**: bearish, neutral or bullish. It follows a Markov chain: each extreme regime lasts five steps on
  average (probability of staying 0.8) and from neutral the market leaves to either side with the same probability.
- **Return** of the next step: normal with mean $\mu_s = -5, 0, +5$ bp depending on the current regime and standard
  deviation $\sigma = 20$ bp.
- **Action**: the new position, short (−1), flat (0) or long (+1).
- **Reward**: the net P&L, $r_{t+1} = a_t \cdot \text{ret}_{t+1} - c \cdot |a_t - p_t|$, with a cost $c = 6$ bp per
  unit traded. Going from short to long costs 12 bp.
- **State**: the pair (regime, current position). The position has to be in the state: with costs, the best action
  depends on where you are. That makes 3 × 3 = 9 states.

```mermaid
stateDiagram-v2
  Bearish --> Neutral: 0.2
  Neutral --> Bearish: 0.1
  Neutral --> Bullish: 0.1
  Bullish --> Neutral: 0.2
```

The regime is observable, as if the agent had a perfect indicator of the state of the market. It is a strong
simplification (in reality it is estimated with noise, lesson 2.2.7), but it isolates what matters here: how a
persistent signal and costs interact.

```python
import numpy as np
import pandas as pd

P = np.array([[0.8, 0.2, 0.0],    # from bearish
              [0.1, 0.8, 0.1],    # from neutral
              [0.0, 0.2, 0.8]])   # from bullish
MU = np.array([-5.0, 0.0, 5.0])   # expected return, bp/step
SIGMA, COSTE, GAMMA = 20.0, 6.0, 0.95
POS = np.array([-1.0, 0.0, 1.0])  # short, flat, long
REG = ["bajista", "neutro", "alcista"]

def recompensas(pos):
    """R[s, p, a]: expected reward in regime s with
    position p when moving to position a."""
    nueva, actual = pos[None, None, :], pos[None, :, None]
    return (MU[:, None, None] * nueva
            - COSTE * np.abs(nueva - actual))

R = recompensas(POS)
w, v = np.linalg.eig(P.T)
estac = np.real(v[:, np.argmin(np.abs(w - 1))])
estac /= estac.sum()
print("stationary distribution:", np.round(estac, 3))
print("mean duration of an extreme regime:",
      round(1 / (1 - P[2, 2]), 1), "steps")
```
```output
stationary distribution: [0.25 0.5  0.25]
mean duration of an extreme regime: 5.0 steps
```

In the long run, the market spends half the time in neutral and a quarter in each extreme. A bullish regime lasts five
steps on average: if the agent goes long when it starts, it expects to earn about 25 bp before it changes, with an
entry cost of 6. The question is whether an algorithm discovers this, and to know that you need the answer.

## The optimal policy: value iteration

The value of a policy $\pi$ in a state is the discounted sum of rewards expected from following it from there. The
optimal action-value function $Q^*(s, a)$ (what you expect to earn if you do $a$ in $s$ and act optimally afterward)
satisfies the **Bellman equation**:

$$
Q^*(s, a) = R(s, a) + \gamma \sum_{s'} P(s' \mid s, a) \max_{a'} Q^*(s', a')
$$

The optimal policy is to choose in each state the action with the largest $Q^*$. When $P$ and $R$ are known, as here,
the equation is solved by applying it over and over starting from any initial table: this is **value iteration**,
which converges because each pass shrinks the error by a factor $\gamma$. In this MDP, the next state is (new regime,
chosen position), so $\max_{a'} Q^*(s', a')$ is the value $V^*$ of the next regime with position $a$.

```python
def iteracion_valor(R, pos=POS, tol=1e-10):
    V = np.zeros((3, len(pos)))
    for it in range(1, 100_000):
        cont = np.einsum("st,ta->sa", P, V)  # E[V(s', a) | s]
        Q = R + GAMMA * cont[:, None, :]
        V_nuevo = Q.max(axis=2)
        if np.abs(V_nuevo - V).max() < tol:
            return Q, it
        V = V_nuevo

Q_opt, iters = iteracion_valor(R)
pi_opt = Q_opt.argmax(axis=2)  # index of the new position
print(f"converges in {iters} iterations")
pd.DataFrame(POS[pi_opt], index=REG,
             columns=["desde corto", "desde fuera", "desde largo"])
```
```output
converges in 463 iterations
         desde corto  desde fuera  desde largo
bajista         -1.0         -1.0         -1.0
neutro          -1.0          0.0          1.0
alcista          1.0          1.0          1.0
```

The optimal policy is easy to read and it is not the obvious one. In a bearish regime, short from anywhere; in a
bullish one, long from anywhere; in **neutral, keep whatever you have**. The mean in neutral is zero, so closing only
costs, and from neutral the market goes back to the previous regime or to the opposite one with the same probability.
It is the hysteresis that costs impose: a zone where you do not trade (lesson 2.4.7 measured it in real strategies;
lesson 4.3.1, in execution).

To compare policies you do not need to simulate. A tabular policy turns the MDP into a Markov chain over the 9 states;
its stationary distribution says how much time is spent in each state, and the long-run mean reward is the mean of the
expected rewards weighted by that time. It is an exact figure, with no sampling noise.

```python
def recompensa_media(pi, pos=POS):
    """Long-run bp/step of a tabular policy pi[s, p]."""
    n = len(pos)
    T = np.zeros((3 * n, 3 * n))
    r = np.zeros(3 * n)
    for s in range(3):
        for p in range(n):
            a = pi[s, p]
            r[s * n + p] = (MU[s] * pos[a]
                            - COSTE * abs(pos[a] - pos[p]))
            T[s * n + p, np.arange(3) * n + a] = P[s]
    w, v = np.linalg.eig(T.T)
    d = np.real(v[:, np.argmin(np.abs(w - 1))])
    return float(d @ r / d.sum())

sin_costes = np.tile([[0], [1], [2]], (1, 3))  # sign of the mean
miope = R.argmax(axis=2)                        # current step only
ref = {"optimal (value iteration)": recompensa_media(pi_opt),
       "signal ignoring costs": recompensa_media(sin_costes),
       "myopic (one step)": recompensa_media(miope)}
for k, val in ref.items():
    print(f"{k:<30}{abs(val):5.2f} bp/step")
```
```output
optimal (value iteration)      1.90 bp/step
signal ignoring costs          1.30 bp/step
myopic (one step)              0.00 bp/step
```

| Policy | Rule | bp per step |
|---|---|---|
| Optimal | Bellman with costs | 1.90 |
| Signal without costs | position = sign of the mean, flat in neutral | 1.30 |
| Myopic | the best for the current step | 0.00 |

The rule that ignores costs loses a third of the optimum by closing in every neutral regime. The myopic one never
trades: going long in a bullish regime gives 5 − 6 = −1 bp in the first step, and only that step counts. It is the
mistake of judging a decision by its first period, very common when designing rules with costs. The `abs()` in the
`print` only avoids the “−0.00” from rounding the myopic result.

What if the position could take any value between −1 and 1? SAC, further down, works with continuous actions.
Repeating value iteration with 21 possible positions answers whether the optimum changes:

```python
pos_fina = np.round(np.linspace(-1, 1, 21), 2)
Q_fina, _ = iteracion_valor(recompensas(pos_fina), pos_fina)
pi_fina = Q_fina.argmax(axis=2)
print("21-position grid:",
      round(recompensa_media(pi_fina, pos_fina), 2), "bp/step")
print("in neutral, from 0.3 it goes to", pos_fina[pi_fina[1, 13]])
```
```output
21-position grid: 1.9 bp/step
in neutral, from 0.3 it goes to 0.3
```

Same value, 1.90 bp per step. With the reward linear in the position, the extremes dominate: in bearish and bullish,
full position; in neutral, keep whatever position you have. Intermediate positions add nothing. So the optimum of the
continuous version is the same, and any difference in SAC will belong to the agent, not to the problem.

## Q-learning: learning without knowing the model

Value iteration needs $P$ and $R$. In a market they are not known: only transitions are observed. **Q-learning**
(Watkins, 1989) learns $Q^*$ from each observed transition $(s, a, r, s')$ with an update that mimics Bellman using a
sample instead of the expectation:

$$
Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]
$$

The term in brackets is the temporal-difference error: what the table expected against what it has seen plus what it
expects from the next state. It converges to $Q^*$ if every (state, action) pair is visited infinitely often and the
step $\alpha$ decreases slowly enough. So that all of them are visited, the agent explores: with probability
$\varepsilon = 0.1$ it chooses an action at random and, otherwise, the best one according to its table (an
$\varepsilon$-greedy policy). The step is $\alpha = n^{-0.7}$, with $n$ the visits to that pair.

```python
rng = np.random.default_rng(0)
PASOS = 300_000
explora = rng.random(PASOS) < 0.1
azar = rng.integers(0, 3, PASOS)
ruido = rng.standard_normal(PASOS)
u = rng.random(PASOS)
acum = P.cumsum(axis=1)

Q = np.zeros((3, 3, 3))
visitas = np.zeros((3, 3, 3))
s, p = 1, 1
idx = np.indices((3, 3))
curva = []
for k in range(PASOS):
    a = azar[k] if explora[k] else int(Q[s, p].argmax())
    r = (POS[a] * (MU[s] + SIGMA * ruido[k])
         - COSTE * abs(POS[a] - POS[p]))
    s2 = int(np.searchsorted(acum[s], u[k], side="right"))
    visitas[s, p, a] += 1
    alfa = visitas[s, p, a] ** -0.7
    Q[s, p, a] += alfa * (r + GAMMA * Q[s2, a].max() - Q[s, p, a])
    s, p = s2, a
    if (k + 1) % 10_000 == 0:
        pi = Q.argmax(axis=2)
        err = np.abs(Q - Q_opt)
        curva.append({
            "pasos": k + 1, "aciertos": int((pi == pi_opt).sum()),
            "pb/paso": round(recompensa_media(pi), 2) + 0.0,
            "error Q máx.": round(err.max(), 1),
            "error Q óptima": round(err[idx[0], idx[1],
                                        pi_opt].max(), 1)})
curva = pd.DataFrame(curva).set_index("pasos")
pi_q = Q.argmax(axis=2)
curva.iloc[[0, 1, 3, 5, 9, 14, 19, 29]]
```
```output
        aciertos  pb/paso  error Q máx.  error Q óptima
pasos
10000          1     0.00          59.0            59.0
20000          2     0.00          55.5            55.5
40000          3     0.65          54.5            54.5
60000          4     0.65          48.2            48.2
100000         6     0.95          43.1            43.1
150000         9     1.90          29.3            17.2
200000         9     1.90          28.2            14.4
300000         9     1.90          27.3             9.7
```

*Fig. — Exact mean reward of Q-learning's greedy policy every 10,000 steps, with the number of states matching the optimum: the policy reaches the optimum at 140,000 steps.* Step line from 0 to 300,000 steps: 0 bp per step until 20,000, 0.65 from 30,000 (3 of 9 states), 0.95 from 70,000 (6 of 9 at 100,000), 1.60 from 120,000 (8 of 9) and 1.90, the exact optimum, from 140,000 (highlighted), when all 9 states match.

The table and the figure tell two different convergence stories. The **policy** is right in all 9 states by 140,000
steps (the figure has the 30 points; the table, eight) and from then on it earns the exact optimum, 1.90 bp per step.
The **values** move much more slowly: the maximum error of the table is still 27.3 bp at the end, and that of the
optimal actions drops to 9.7 bp, but does not reach zero either. The actions the greedy policy does not choose are only
tried in the 10% of exploration, and their value is estimated with few samples of a reward with a 20 bp standard
deviation. To decide, it is enough that in each state the good action has the highest value.

It takes hundreds of thousands of steps for a 9-state problem with a huge signal. A step here is a session, a minute
or a trade depending on the problem; 140,000 sessions are more than 550 years. That is the first reason RL in trading
is almost always trained in simulators, and the reason the simulator has to be good (lesson 5.9.2).

## A Gymnasium environment with costs in the reward

To use algorithms that are already written, the problem is packaged with the standard interface of **Gymnasium**
(1.4, the continuation of OpenAI Gym maintained by the Farama Foundation): a class with an observation space, an action
space, `reset()` to start an episode and `step(action)` to advance one step, which returns the observation, the reward,
whether the episode has ended (`terminated`, an end of the problem) or been cut off (`truncated`, a time limit) and an
info dictionary.

Design decisions, which in RL weigh as much as the algorithm:

- **Observation**: the regime in *one-hot* encoding (three columns) and the current position. A neural network learns
  better with the regime as three 0-and-1 inputs than as a number 0, 1, 2 that suggests an order.
- **Costs inside the reward**: the agent optimizes what it is rewarded for. If the cost is not in the reward, it does
  not exist for the agent (lesson 5.9.3 measures this).
- **Scale**: the reward is the P&L divided by σ = 20 bp. Stable-Baselines3 networks and optimizers have default
  hyperparameters designed for rewards of order 1; with the P&L in bp, the critic's gradients are hundreds of times
  larger.
- **250-step episodes**, cut off with `truncated` and not with `terminated`: the market does not end, it just stops
  being observed. With `terminated`, the algorithm would believe there is no future value after step 250.
- **Environment randomness** with `self.np_random`, which `reset(seed=…)` seeds: that way each episode is
  reproducible.

```python
import gymnasium as gym
from gymnasium import spaces
from gymnasium.utils.env_checker import check_env

class RegimenEnv(gym.Env):
    """Regime MDP: observes the regime (one-hot) and the position;
    the action is the new position; the reward, the net P&L."""

    def __init__(self, continua=False, pasos=250):
        self.continua, self.pasos = continua, pasos
        self.observation_space = spaces.Box(-1.0, 1.0, (4,),
                                            np.float32)
        if continua:
            self.action_space = spaces.Box(-1.0, 1.0, (1,),
                                           np.float32)
        else:
            self.action_space = spaces.Discrete(3)

    def _obs(self):
        o = np.zeros(4, np.float32)
        o[self.s], o[3] = 1.0, self.pos
        return o

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        u = self.np_random.random()
        self.s = int(np.searchsorted(estac.cumsum(), u,
                                     side="right"))
        self.pos, self.t = 0.0, 0
        return self._obs(), {}

    def step(self, action):
        if self.continua:
            nueva = float(np.clip(action[0], -1.0, 1.0))
        else:
            nueva = float(POS[int(action)])
        ret = MU[self.s] + SIGMA * self.np_random.standard_normal()
        pnl = nueva * ret - COSTE * abs(nueva - self.pos)
        u = self.np_random.random()
        self.s = int(np.searchsorted(acum[self.s], u, side="right"))
        self.pos, self.t = nueva, self.t + 1
        truncado = self.t >= self.pasos
        return self._obs(), pnl / SIGMA, False, truncado, {"pnl": pnl}

check_env(RegimenEnv(), skip_render_check=True)
check_env(RegimenEnv(continua=True), skip_render_check=True)
print("check_env: no errors in either version")
env = RegimenEnv()
obs, _ = env.reset(seed=1)
print(obs, env.observation_space, env.action_space)
```
```output
check_env: no errors in either version
[0. 1. 0. 0.] Box(-1.0, 1.0, (4,), float32) Discrete(3)
```

`check_env` checks the interface: that `reset` and `step` return what they should, that the observations are inside
their space and of their type (`float32`), that the seed makes episodes reproducible. `skip_render_check=True` skips
the test of the render modes, which this environment does not have (without it, it warns that the environment was not
created with `gymnasium.make`). What `check_env` does not check is the economics: a cost with the wrong sign, the wrong
step's return (today's instead of tomorrow's, a leak like those in 5.4.1) or a position that is not stored pass the
test all the same. Those are caught by comparing with a known solution, as here, or with no-signal controls.

## Policy gradient and PPO

Q-learning learns values and derives the policy from them. **Policy gradient** methods parametrize the policy
directly, $\pi_\theta(a \mid s)$ (a network that gives probabilities for each action), and climb the gradient of the
expected reward:

$$
\nabla_\theta J(\theta) = \mathbb{E}\left[ \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \hat{A}_t \right]
$$

where $\hat{A}_t$ is the **advantage**: how much better the action $a_t$ turned out than the critic (a second network
that estimates the value of the state) expected. Actions that turned out better than expected become more likely. The
problem is that a large gradient step can wreck the policy at once, and with noisy rewards that happens often.

**PPO** (*Proximal Policy Optimization*, Schulman et al., 2017) limits how much the policy can change in each update.
With $\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$, the ratio between the new policy
and the one that collected the data, it maximizes

$$
L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[ \min\left( \rho_t \hat{A}_t,\ \operatorname{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon)\, \hat{A}_t \right) \right]
$$

with $\epsilon = 0.2$: beyond a 20% change in the probability of an action, the objective stops rewarding the change.
It is the default algorithm in much RL work because of its stability. It works with discrete and continuous actions;
here, with the three positions.

The configuration: 8 copies of the environment in parallel (`make_vec_env`, each with its own seed), 256 steps per copy
before each update, a network of two layers of 32 neurons, the same discount as the MDP and `seed=0` with
`device="cpu"` and deterministic PyTorch algorithms. 60,000 steps in three blocks, and after each block the agent's
deterministic action in the 9 states is read and evaluated with the exact formula.

```python
import torch
from stable_baselines3 import PPO, SAC
from stable_baselines3.common.env_util import make_vec_env

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)

OBS9 = np.zeros((9, 4), np.float32)  # the 9 states
for s in range(3):
    for p in range(3):
        OBS9[3 * s + p, s], OBS9[3 * s + p, 3] = 1.0, POS[p]

def tabla_politica(modelo):
    """The agent's deterministic action in the 9 states."""
    a, _ = modelo.predict(OBS9, deterministic=True)
    return np.asarray(a).reshape(3, 3)

vec = make_vec_env(RegimenEnv, n_envs=8, seed=0)
ppo = PPO("MlpPolicy", vec, gamma=GAMMA, n_steps=256,
          batch_size=256, learning_rate=1e-3,
          policy_kwargs={"net_arch": [32, 32]},
          seed=0, device="cpu", verbose=0)
for bloque in range(3):
    ppo.learn(20_000, reset_num_timesteps=bloque == 0)
    pi_ppo = tabla_politica(ppo)
    print(f"{20_000 * (bloque + 1):>6} steps: "
          f"{int((pi_ppo == pi_opt).sum())}/9 match the optimum, "
          f"{recompensa_media(pi_ppo):.2f} bp/step")
pd.DataFrame(POS[pi_ppo], index=REG,
             columns=["desde corto", "desde fuera", "desde largo"])
```
```output
 20000 steps: 9/9 match the optimum, 1.90 bp/step
 40000 steps: 8/9 match the optimum, 1.90 bp/step
 60000 steps: 8/9 match the optimum, 1.90 bp/step
         desde corto  desde fuera  desde largo
bajista         -1.0         -1.0         -1.0
neutro          -1.0         -1.0          1.0
alcista          1.0          1.0          1.0
```

PPO finds the optimal policy by 20,000 steps and then changes the action in one state: in neutral and flat, it opens a
short. The long-run mean reward does not move (1.90 bp per step) because that state is not part of the policy's cycle:
once in, you are never flat again. It is only visited at the start of each episode, so the agent has very few samples
of it and the network decides it by proximity to the neighboring states. It is a general warning: an agent is reliable
where it has spent time and arbitrary where it has not, and the rare states of a market (a crisis, a gap) are exactly
the ones that matter most.

## SAC: continuous actions and entropy

With a continuous position between −1 and 1, the usual choice is **SAC** (*Soft Actor-Critic*, Haarnoja et al., 2018):
an *off-policy* method (it reuses old transitions stored in a buffer, which makes it more sample-efficient than PPO)
that maximizes the reward plus a bonus for the entropy of the policy:

$$
J(\pi) = \sum_t \mathbb{E}\left[ r_t + \alpha\, \mathcal{H}\big(\pi(\cdot \mid s_t)\big) \right]
$$

The entropy term keeps exploration going, and $\alpha$ sets how much. Stable-Baselines3 tunes it automatically
(`ent_coef="auto"`) to reach a target entropy. In the tests prior to this lesson, with the same budget of 8,000 steps,
the default initial value of $\alpha$ (1.0) left a timid policy, with positions of 0.1 to 0.45 in absolute value where
the optimum calls for 1: the entropy weighed more than rewards of a few tenths. With `"auto_0.01"` (automatic tuning
starting at 0.01) it gets much closer. And with the reward divided by 100 (as a percentage) instead of by σ, SAC
learned nothing: positions close to zero. They are the two knobs that change the result the most, and neither comes
from the financial problem: it is algorithm engineering.

```python
vec_c = make_vec_env(lambda: RegimenEnv(continua=True), n_envs=4,
                     seed=0)
sac = SAC("MlpPolicy", vec_c, gamma=GAMMA, learning_starts=1000,
          batch_size=128, ent_coef="auto_0.01",
          policy_kwargs={"net_arch": [32, 32]},
          seed=0, device="cpu", verbose=0)
sac.learn(8_000)
a_sac = sac.predict(OBS9, deterministic=True)[0].reshape(3, 3)
pd.DataFrame(a_sac, index=REG,
             columns=["desde corto", "desde fuera",
                      "desde largo"]).round(2)
```
```output
         desde corto  desde fuera  desde largo
bajista        -0.92        -0.65        -0.57
neutro         -0.82         0.42         0.95
alcista         0.87         0.96         0.98
```

SAC has learned the shape of the optimal policy: long in bullish, short in bearish and, in neutral, a position that
grows with the one it already had (the hysteresis). But it does not reach the extremes: in bearish it stays between
−0.57 and −0.92 instead of −1. With more steps it would get closer; with 8,000 (4,000 network updates, a few seconds of
CPU), this is what it gives. In production it would be trained longer and with several seeds, and even then the gap
would have to be measured.

## The honest comparison

The exact mean reward only exists for tabular policies. To compare all of them, SAC included, the same 40 episodes of
250 steps (seeds 10,000 to 10,039) are simulated with each policy: same regimes and same returns, so the difference
from the optimum does not depend on the randomness of the paths, only on the decisions. The standard error of that
difference comes from the 40 episodes.

```python
def evalua(politica, episodios=40, pasos=250):
    """Mean bp/step of each episode, same paths for all."""
    res = []
    for e in range(episodios):
        env = RegimenEnv(continua=True, pasos=pasos)
        o, _ = env.reset(seed=10_000 + e)
        tot = 0.0
        for _ in range(pasos):
            a = np.array([politica(o)], np.float32)
            o, _, _, _, info = env.step(a)
            tot += info["pnl"]
        res.append(tot / pasos)
    return np.array(res)

def de_tabla(pi):
    return lambda o: POS[pi[int(o[:3].argmax()), int(o[3]) + 1]]

def de_sac(o):
    return float(sac.predict(o, deterministic=True)[0][0])

politicas = {"óptima": de_tabla(pi_opt), "Q-learning": de_tabla(pi_q),
             "PPO": de_tabla(pi_ppo), "SAC": de_sac,
             "sin costes": de_tabla(sin_costes),
             "miope": de_tabla(miope)}
res = {k: evalua(f) for k, f in politicas.items()}
filas = []
for k, x in res.items():
    d = x - res["óptima"]
    filas.append({"política": k, "pb/paso": round(x.mean(), 2),
                  "frente al óptimo": round(d.mean(), 2),
                  "error típico": round(d.std(ddof=1) / 40**0.5, 2)})
pd.DataFrame(filas).set_index("política")
```
```output
            pb/paso  frente al óptimo  error típico
política
óptima         1.99              0.00          0.00
Q-learning     1.99              0.00          0.00
PPO            1.93             -0.06          0.02
SAC            1.55             -0.44          0.13
sin costes     1.28             -0.71          0.15
miope          0.00             -1.99          0.24
```

The header figure summarizes the table. On these 40 paths the optimum earns 1.99 bp per step (somewhat more than the
exact 1.90: it is a sample). Q-learning, with the optimal policy in all 9 states, matches it to the cent. PPO ends up
0.06 bp below, with a standard error of 0.02: it is its mistake in the initial state, which does count in short
episodes. SAC ends up 0.44 bp behind (standard error 0.13), 22% below. The rule without costs loses 0.71 and the
myopic one, everything.

| Agent | Steps | Result vs the optimum | What is missing |
|---|---|---|---|
| Tabular Q-learning | 300,000 | equal | nothing in the policy; the values, far off |
| PPO | 60,000 | −0.06 bp (−3%) | a rare state poorly learned |
| SAC | 8,000 | −0.44 bp (−22%) | budget; it does not reach full positions |

Three lessons that hold for the whole module. First: without the optimum you would not know that SAC leaves 22% on
the table or that PPO fails in one state; in a market there is no optimum, so you have to build references
(analytical, as in 5.9.2) and no-signal controls (as in 5.9.3). Second: the algorithms are sensitive to decisions that
have nothing financial about them (reward scale, entropy, budget), and a result is only worth as much as its
configuration. Third: even in a 9-state problem with a signal twenty times better than a market's, tens of thousands
of steps are needed.

<aside class="callout" data-kind="riesgo">

This environment is stationary: the regime probabilities and the means never change. A market does change (lesson
5.10.2), and an agent trained on a past that does not repeat learns the past. With a real signal with an IC of a few
hundredths and a few thousand sessions, the risk that the agent learns the noise of the sample is the rule, not the
exception. It is the topic of lesson 5.9.3.

</aside>

## Exercises

1. Sweep the cost per unit traded: 6, 12, 18 and 24 bp. For each one, compute the optimal policy, in how many of the
   9 states it changes position and its mean reward, alongside that of the rule without costs. When does trading stop
   paying off?

**Solution.**

```python
for c in (6.0, 12.0, 18.0, 24.0):
    COSTE = c  # recompensas() and recompensa_media() read it
    pic = iteracion_valor(recompensas(POS))[0].argmax(axis=2)
    opera = int((POS[pic] != POS[None, :]).sum())
    print(f"cost {c:4.0f} bp: changes in {opera}/9 states, "
          f"optimal {abs(recompensa_media(pic)):5.2f}, "
          f"no costs {recompensa_media(sin_costes):6.2f} bp/step")
COSTE = 6.0  # restored for the rest of the lesson
```
```output
cost    6 bp: changes in 4/9 states, optimal  1.90, no costs   1.30 bp/step
cost   12 bp: changes in 4/9 states, optimal  1.30, no costs   0.10 bp/step
cost   18 bp: changes in 4/9 states, optimal  0.70, no costs  -1.10 bp/step
cost   24 bp: changes in 0/9 states, optimal  0.00, no costs  -2.30 bp/step
```

Up to 18 bp the optimal policy does not change: it trades in the same 4 states (entering or reversing when an extreme
regime arrives) and holds in the rest. What drops is what it earns: 1.90, 1.30 and 0.70 bp per step, 0.60 bp less for
every 6 bp of cost. That slope is its turnover, 0.1 units per step; the gross is always 2.5 bp (half the time in an
extreme regime, earning 5). At 24 bp it stops trading altogether: entering costs almost what it expects to earn over
the whole regime, and reversing, twice as much. The rule without costs turns over twice as much (0.2 units per step)
and loses money from 18 bp. Given a signal and costs, trading frequency is not a free parameter: Bellman decides it.

2. Repeat the Q-learning with a constant step $\alpha = 0.05$ instead of $n^{-0.7}$ and compare the matches and the
   mean reward of its policy at 50,000, 150,000 and 300,000 steps.

**Solution.**

```python
Qc = np.zeros((3, 3, 3))
s, p = 1, 1
for k in range(PASOS):
    a = azar[k] if explora[k] else int(Qc[s, p].argmax())
    r = (POS[a] * (MU[s] + SIGMA * ruido[k])
         - COSTE * abs(POS[a] - POS[p]))
    s2 = int(np.searchsorted(acum[s], u[k], side="right"))
    Qc[s, p, a] += 0.05 * (r + GAMMA * Qc[s2, a].max() - Qc[s, p, a])
    s, p = s2, a
    if (k + 1) in (50_000, 150_000, 300_000):
        pi = Qc.argmax(axis=2)
        print(f"{k + 1:>7}: {int((pi == pi_opt).sum())}/9, "
              f"{recompensa_media(pi):.2f} bp/step")
```
```output
  50000: 4/9, 0.95 bp/step
 150000: 5/9, 0.65 bp/step
 300000: 8/9, 1.81 bp/step
```

With a constant step, the table never stops moving: each noisy reward pushes it by 5%, so the values oscillate around
Q* with an amplitude that depends on α and on the noise, instead of converging. Here it is right in 4 of 9 states at
50,000 steps, in 5 at 150,000 (and earns less than at 50,000: 0.65 vs 0.95 bp per step) and in 8 at 300,000, with
1.81 bp per step: it never reaches 9, because in the states where two actions have similar values the noise swaps the
order again and again. It is the flip side of what in a non-stationary market would be a virtue: a constant step
forgets the past and follows the new regime.

3. A colleague writes the `step` of their environment with the reward `nueva * ret` and subtracts the costs afterward,
   when computing the backtest P&L. Their agent trades at every step. What has happened and what would `check_env` have
   detected?

**Solution.**

The agent has optimized the reward it was given, in which trading costs nothing: it does not care about reversing the
position every step, and in this MDP the optimal policy without costs is the signal without hysteresis (closing in
neutral), which with costs earns 1.30 instead of 1.90 bp per step; with a higher cost, it would lose money.
`check_env` would have detected nothing: the interface is correct. Costs have to be inside the reward, and the result
of training must always be evaluated with the full net P&L, never with the training reward (lesson 5.9.3, where a
similar mistake in the costs produces an agent that looks good and is not).

## Self-assessment

1. In the lesson's MDP, with a cost of 6 bp per unit traded, the myopic policy (the one that maximizes only the current step's reward) earns 0 bp per step. Why?

   - A. Because going long in a bullish regime gives 5 − 6 = −1 bp in the first step, so it never enters, even though the regime lasts five steps on average
   - B. Because the regime signal carries no information
   - C. Because the discount γ = 0.95 cancels future rewards
   - D. Because the myopic policy trades too much

   *Answer A: Because going long in a bullish regime gives 5 − 6 = −1 bp in the first step, so it never enters, even though the regime lasts five steps on average.* The myopic policy compares only the first step, where the cost weighs more than the gain. The optimal one counts the following steps of the regime (Bellman) and enters: 1.90 bp per step.

2. The optimal policy keeps the position when the regime turns neutral; the rule “position = sign of the mean” closes it. What does the former gain?

   - A. Nothing: both earn 1.90 bp per step
   - B. It earns more because the expected return in neutral is positive
   - C. It reduces variance without changing the mean
   - D. It saves round-trip costs: in neutral the mean is zero, so closing only costs, and from neutral the market goes back to the previous regime or to the opposite one with the same probability. 1.90 vs 1.30 bp per step

   *Answer D: It saves round-trip costs: in neutral the mean is zero, so closing only costs, and from neutral the market goes back to the previous regime or to the opposite one with the same probability. 1.90 vs 1.30 bp per step.* It is the hysteresis that costs impose: a no-trade zone. The policy that ignores costs turns over more and loses 0.60 bp per step, a third of the optimum.

3. After 300,000 steps, Q-learning chooses the optimal action in all 9 states, but the maximum error of its Q table against Q* is still large. Which reading is correct?

   - A. The table is wrong: the training has to be repeated
   - B. Q-learning never converges on problems with noise
   - C. It is enough for the order of the actions to be correct in each state; little-explored actions have poorly estimated values, and that does not change the decision
   - D. The error is due to the discount γ

   *Answer C: It is enough for the order of the actions to be correct in each state; little-explored actions have poorly estimated values, and that does not change the decision.* The policy converges before the values. With ε = 0.1, worse actions are rarely visited and their Q learns slowly; to decide, all that matters is which one is the largest in each state.

4. PPO matches the optimum in 8 of 9 states and its long-run mean reward is the same, 1.90 bp per step. On the 40 test paths it ends up 0.06 bp below. Where does that difference come from?

   - A. From the randomness of the paths: it is not a real difference
   - B. From the neural network, which rounds the actions
   - C. From the state it gets wrong (neutral and flat), which is only visited at the start of each episode: it carries no weight in the long run, but in 250-step episodes it does count
   - D. From PPO using a different discount

   *Answer C: From the state it gets wrong (neutral and flat), which is only visited at the start of each episode: it carries no weight in the long run, but in 250-step episodes it does count.* A state the optimal policy almost never visits gives few samples to learn from. The exact long-run evaluation does not see it; the per-episode evaluation does. You have to look at both.

5. SAC on the continuous version of the problem ends up at 1.55 bp per step vs the optimum's 1.99 on the same paths. Which conclusion is honest?

   - A. SAC is useless for trading
   - B. With 8,000 steps and these hyperparameters, SAC learns the shape of the policy but does not reach the extremes; the result depends on the budget, the reward scale and the entropy coefficient, and you only know how far off it is because the optimum is known
   - C. SAC is better than PPO because it uses continuous actions
   - D. The continuous version has a different optimum

   *Answer B: With 8,000 steps and these hyperparameters, SAC learns the shape of the policy but does not reach the extremes; the result depends on the budget, the reward scale and the entropy coefficient, and you only know how far off it is because the optimum is known.* Value iteration with 21 positions gives the same optimum as with three, so the gap belongs to the agent. Without a reference optimum, that gap would be invisible.

## Key points

- An MDP is (states, actions, transitions, reward, discount). In trading, the state includes the position, because with costs the best action depends on where you are.
- In the lesson's MDP (three regimes, μ = ±5 bp, σ = 20 bp, a 6 bp cost), value iteration gives the exact optimal policy: 1.90 bp per step, vs 1.30 for the rule that ignores costs and 0 for the myopic one.
- Tabular Q-learning recovers the optimal policy in all 9 states by 140,000 steps; its values are still far from Q*. The policy converges before the values.
- A `gymnasium.Env` environment is `reset`, `step` and two spaces. Costs go inside the reward; `check_env` checks the interface, not the economics.
- PPO (60,000 steps) gets 8 of 9 states right: the one it misses is almost never visited. SAC (8,000 steps) learns the shape but stays at 1.55 bp per step. The reward scale and the entropy decide whether it learns.
- Knowing the optimum is what makes it possible to measure an agent. In a market there is no optimum, which is why lessons 5.9.2 and 5.9.3 always compare with analytical references and with no-signal controls.

## Resources

### Research

- [Finite-State Machines for Quantitative Trading Execution in Python](https://pythonparatrading.com/en/finite-state-machines-trading-python/) · Research-0029 · 47 min

### Documentation

- [Gymnasium: create a custom environment](https://gymnasium.farama.org/introduction/create_custom_env/) · gymnasium.farama.org
- [Gymnasium: utilities, including check_env](https://gymnasium.farama.org/api/utils/) · gymnasium.farama.org
- [Stable-Baselines3: PPO](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html) · stable-baselines3.readthedocs.io
- [Stable-Baselines3: SAC](https://stable-baselines3.readthedocs.io/en/master/modules/sac.html) · stable-baselines3.readthedocs.io
- [Schulman, J. et al. (2017): Proximal Policy Optimization Algorithms. arXiv:1707.06347](https://arxiv.org/abs/1707.06347) · arxiv.org

### Books

- [Sutton, R. S. and Barto, A. G. (2018): Reinforcement Learning: An Introduction, 2nd ed. MIT Press (the full text is also on incompleteideas.net, the authors' website)](https://mitpress.mit.edu/9780262039246/reinforcement-learning/) · mitpress.mit.edu

Previous lesson: [5.8.5 · Lab: foundation models vs GARCH, HAR and LightGBM](https://pythonparatrading.com/en/python-trading-course/level-5/foundation-models-vs-garch-lightgbm-lab/)

Next lesson: [5.9.2 · RL for execution, market making and dynamic hedging](https://pythonparatrading.com/en/python-trading-course/level-5/rl-execution-market-making-hedging/)