MDP, Q-learning, PPO and Gymnasium environments.

From backtest to sequential decision

Everything level 5 has done so far is supervised learning: a table of features, a label and a model that predicts the label. The trading decision came afterward, as a fixed rule on top of the prediction (buy the top decile, lesson 5.4.4). Reinforcement learning (RL) brings the two together: an agent observes the state, chooses an action, the environment returns a reward and a new state, and the agent learns the decision rule that maximizes the sum of future rewards. There is no label: only the consequence of each action.

flowchart LR
  A[Agent] -->|action| E[Environment]
  E -->|reward| A
  E -->|new state| A

That shape fits trading problems where today's action changes tomorrow's problem: an open position that costs money to close, an inventory that has to be liquidated (lesson 4.3.1), the quotes of a market maker that accumulate inventory (lesson 2.3.6). The price of the flexibility is high: the agent learns by trial and error, with noisy rewards, and nothing guarantees that what it learns is optimal. That is why this lesson starts with a problem whose exact answer can be computed, and only then lets the algorithms look for it.

A trading problem as an MDP

A Markov decision process (MDP) is a tuple $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$: states, actions, transition probabilities $P(s' \mid s, a)$, expected reward $R(s, a)$ and discount factor $\gamma \in [0, 1)$. “Markov” means that the current state summarizes everything about the past that matters for what comes next.

The lesson's problem:

• Market regime: bearish, neutral or bullish. It follows a Markov chain: each extreme regime lasts five steps on average (probability of staying 0.8) and from neutral the market leaves to either side with the same probability.
• Return of the next step: normal with mean $\mu_s = -5, 0, +5$ bp depending on the current regime and standard deviation $\sigma = 20$ bp.
• Action: the new position, short (−1), flat (0) or long (+1).
• Reward: the net P&L, $r_{t+1} = a_t \cdot \text{ret}_{t+1} - c \cdot |a_t - p_t|$, with a cost $c = 6$ bp per unit traded. Going from short to long costs 12 bp.
• State: the pair (regime, current position). The position has to be in the state: with costs, the best action depends on where you are. That makes 3 × 3 = 9 states.

stateDiagram-v2
  Bearish --> Neutral: 0.2
  Neutral --> Bearish: 0.1
  Neutral --> Bullish: 0.1
  Bullish --> Neutral: 0.2

The regime is observable, as if the agent had a perfect indicator of the state of the market. It is a strong simplification (in reality it is estimated with noise, lesson 2.2.7), but it isolates what matters here: how a persistent signal and costs interact.

import numpy as np
import pandas as pd

P = np.array([[0.8, 0.2, 0.0],    # from bearish
              [0.1, 0.8, 0.1],    # from neutral
              [0.0, 0.2, 0.8]])   # from bullish
MU = np.array([-5.0, 0.0, 5.0])   # expected return, bp/step
SIGMA, COSTE, GAMMA = 20.0, 6.0, 0.95
POS = np.array([-1.0, 0.0, 1.0])  # short, flat, long
REG = ["bajista", "neutro", "alcista"]

def recompensas(pos):
    """R[s, p, a]: expected reward in regime s with
    position p when moving to position a."""
    nueva, actual = pos[None, None, :], pos[None, :, None]
    return (MU[:, None, None] * nueva
            - COSTE * np.abs(nueva - actual))

R = recompensas(POS)
w, v = np.linalg.eig(P.T)
estac = np.real(v[:, np.argmin(np.abs(w - 1))])
estac /= estac.sum()
print("stationary distribution:", np.round(estac, 3))
print("mean duration of an extreme regime:",
      round(1 / (1 - P[2, 2]), 1), "steps")

stationary distribution: [0.25 0.5  0.25]
mean duration of an extreme regime: 5.0 steps

In the long run, the market spends half the time in neutral and a quarter in each extreme. A bullish regime lasts five steps on average: if the agent goes long when it starts, it expects to earn about 25 bp before it changes, with an entry cost of 6. The question is whether an algorithm discovers this, and to know that you need the answer.

The optimal policy: value iteration

The value of a policy $\pi$ in a state is the discounted sum of rewards expected from following it from there. The optimal action-value function $Q^*(s, a)$ (what you expect to earn if you do $a$ in $s$ and act optimally afterward) satisfies the Bellman equation:

The optimal policy is to choose in each state the action with the largest $Q^*$. When $P$ and $R$ are known, as here, the equation is solved by applying it over and over starting from any initial table: this is value iteration, which converges because each pass shrinks the error by a factor $\gamma$. In this MDP, the next state is (new regime, chosen position), so $\max_{a'} Q^*(s', a')$ is the value $V^*$ of the next regime with position $a$.

def iteracion_valor(R, pos=POS, tol=1e-10):
    V = np.zeros((3, len(pos)))
    for it in range(1, 100_000):
        cont = np.einsum("st,ta->sa", P, V)  # E[V(s', a) | s]
        Q = R + GAMMA * cont[:, None, :]
        V_nuevo = Q.max(axis=2)
        if np.abs(V_nuevo - V).max() < tol:
            return Q, it
        V = V_nuevo

Q_opt, iters = iteracion_valor(R)
pi_opt = Q_opt.argmax(axis=2)  # index of the new position
print(f"converges in {iters} iterations")
pd.DataFrame(POS[pi_opt], index=REG,
             columns=["desde corto", "desde fuera", "desde largo"])

converges in 463 iterations
         desde corto  desde fuera  desde largo
bajista         -1.0         -1.0         -1.0
neutro          -1.0          0.0          1.0
alcista          1.0          1.0          1.0

The optimal policy is easy to read and it is not the obvious one. In a bearish regime, short from anywhere; in a bullish one, long from anywhere; in neutral, keep whatever you have. The mean in neutral is zero, so closing only costs, and from neutral the market goes back to the previous regime or to the opposite one with the same probability. It is the hysteresis that costs impose: a zone where you do not trade (lesson 2.4.7 measured it in real strategies; lesson 4.3.1, in execution).

To compare policies you do not need to simulate. A tabular policy turns the MDP into a Markov chain over the 9 states; its stationary distribution says how much time is spent in each state, and the long-run mean reward is the mean of the expected rewards weighted by that time. It is an exact figure, with no sampling noise.

def recompensa_media(pi, pos=POS):
    """Long-run bp/step of a tabular policy pi[s, p]."""
    n = len(pos)
    T = np.zeros((3 * n, 3 * n))
    r = np.zeros(3 * n)
    for s in range(3):
        for p in range(n):
            a = pi[s, p]
            r[s * n + p] = (MU[s] * pos[a]
                            - COSTE * abs(pos[a] - pos[p]))
            T[s * n + p, np.arange(3) * n + a] = P[s]
    w, v = np.linalg.eig(T.T)
    d = np.real(v[:, np.argmin(np.abs(w - 1))])
    return float(d @ r / d.sum())

sin_costes = np.tile([[0], [1], [2]], (1, 3))  # sign of the mean
miope = R.argmax(axis=2)                        # current step only
ref = {"optimal (value iteration)": recompensa_media(pi_opt),
       "signal ignoring costs": recompensa_media(sin_costes),
       "myopic (one step)": recompensa_media(miope)}
for k, val in ref.items():
    print(f"{k:<30}{abs(val):5.2f} bp/step")

optimal (value iteration)      1.90 bp/step
signal ignoring costs          1.30 bp/step
myopic (one step)              0.00 bp/step

The rule that ignores costs loses a third of the optimum by closing in every neutral regime. The myopic one never trades: going long in a bullish regime gives 5 − 6 = −1 bp in the first step, and only that step counts. It is the mistake of judging a decision by its first period, very common when designing rules with costs. The abs() in the print only avoids the “−0.00” from rounding the myopic result.

What if the position could take any value between −1 and 1? SAC, further down, works with continuous actions. Repeating value iteration with 21 possible positions answers whether the optimum changes:

pos_fina = np.round(np.linspace(-1, 1, 21), 2)
Q_fina, _ = iteracion_valor(recompensas(pos_fina), pos_fina)
pi_fina = Q_fina.argmax(axis=2)
print("21-position grid:",
      round(recompensa_media(pi_fina, pos_fina), 2), "bp/step")
print("in neutral, from 0.3 it goes to", pos_fina[pi_fina[1, 13]])

21-position grid: 1.9 bp/step
in neutral, from 0.3 it goes to 0.3

Same value, 1.90 bp per step. With the reward linear in the position, the extremes dominate: in bearish and bullish, full position; in neutral, keep whatever position you have. Intermediate positions add nothing. So the optimum of the continuous version is the same, and any difference in SAC will belong to the agent, not to the problem.

Q-learning: learning without knowing the model

Value iteration needs $P$ and $R$. In a market they are not known: only transitions are observed. Q-learning (Watkins, 1989) learns $Q^*$ from each observed transition $(s, a, r, s')$ with an update that mimics Bellman using a sample instead of the expectation:

The term in brackets is the temporal-difference error: what the table expected against what it has seen plus what it expects from the next state. It converges to $Q^*$ if every (state, action) pair is visited infinitely often and the step $\alpha$ decreases slowly enough. So that all of them are visited, the agent explores: with probability $\varepsilon = 0.1$ it chooses an action at random and, otherwise, the best one according to its table (an $\varepsilon$-greedy policy). The step is $\alpha = n^{-0.7}$, with $n$ the visits to that pair.

rng = np.random.default_rng(0)
PASOS = 300_000
explora = rng.random(PASOS) < 0.1
azar = rng.integers(0, 3, PASOS)
ruido = rng.standard_normal(PASOS)
u = rng.random(PASOS)
acum = P.cumsum(axis=1)

Q = np.zeros((3, 3, 3))
visitas = np.zeros((3, 3, 3))
s, p = 1, 1
idx = np.indices((3, 3))
curva = []
for k in range(PASOS):
    a = azar[k] if explora[k] else int(Q[s, p].argmax())
    r = (POS[a] * (MU[s] + SIGMA * ruido[k])
         - COSTE * abs(POS[a] - POS[p]))
    s2 = int(np.searchsorted(acum[s], u[k], side="right"))
    visitas[s, p, a] += 1
    alfa = visitas[s, p, a] ** -0.7
    Q[s, p, a] += alfa * (r + GAMMA * Q[s2, a].max() - Q[s, p, a])
    s, p = s2, a
    if (k + 1) % 10_000 == 0:
        pi = Q.argmax(axis=2)
        err = np.abs(Q - Q_opt)
        curva.append({
            "pasos": k + 1, "aciertos": int((pi == pi_opt).sum()),
            "pb/paso": round(recompensa_media(pi), 2) + 0.0,
            "error Q máx.": round(err.max(), 1),
            "error Q óptima": round(err[idx[0], idx[1],
                                        pi_opt].max(), 1)})
curva = pd.DataFrame(curva).set_index("pasos")
pi_q = Q.argmax(axis=2)
curva.iloc[[0, 1, 3, 5, 9, 14, 19, 29]]

        aciertos  pb/paso  error Q máx.  error Q óptima
pasos
10000          1     0.00          59.0            59.0
20000          2     0.00          55.5            55.5
40000          3     0.65          54.5            54.5
60000          4     0.65          48.2            48.2
100000         6     0.95          43.1            43.1
150000         9     1.90          29.3            17.2
200000         9     1.90          28.2            14.4
300000         9     1.90          27.3             9.7

The table and the figure tell two different convergence stories. The policy is right in all 9 states by 140,000 steps (the figure has the 30 points; the table, eight) and from then on it earns the exact optimum, 1.90 bp per step. The values move much more slowly: the maximum error of the table is still 27.3 bp at the end, and that of the optimal actions drops to 9.7 bp, but does not reach zero either. The actions the greedy policy does not choose are only tried in the 10% of exploration, and their value is estimated with few samples of a reward with a 20 bp standard deviation. To decide, it is enough that in each state the good action has the highest value.

It takes hundreds of thousands of steps for a 9-state problem with a huge signal. A step here is a session, a minute or a trade depending on the problem; 140,000 sessions are more than 550 years. That is the first reason RL in trading is almost always trained in simulators, and the reason the simulator has to be good (lesson 5.9.2).

A Gymnasium environment with costs in the reward

To use algorithms that are already written, the problem is packaged with the standard interface of Gymnasium (1.4, the continuation of OpenAI Gym maintained by the Farama Foundation): a class with an observation space, an action space, reset() to start an episode and step(action) to advance one step, which returns the observation, the reward, whether the episode has ended (terminated, an end of the problem) or been cut off (truncated, a time limit) and an info dictionary.

Design decisions, which in RL weigh as much as the algorithm:

• Observation: the regime in one-hot encoding (three columns) and the current position. A neural network learns better with the regime as three 0-and-1 inputs than as a number 0, 1, 2 that suggests an order.
• Costs inside the reward: the agent optimizes what it is rewarded for. If the cost is not in the reward, it does not exist for the agent (lesson 5.9.3 measures this).
• Scale: the reward is the P&L divided by σ = 20 bp. Stable-Baselines3 networks and optimizers have default hyperparameters designed for rewards of order 1; with the P&L in bp, the critic's gradients are hundreds of times larger.
• 250-step episodes, cut off with truncated and not with terminated: the market does not end, it just stops being observed. With terminated, the algorithm would believe there is no future value after step 250.
• Environment randomness with self.np_random, which reset(seed=…) seeds: that way each episode is reproducible.

import gymnasium as gym
from gymnasium import spaces
from gymnasium.utils.env_checker import check_env

class RegimenEnv(gym.Env):
    """Regime MDP: observes the regime (one-hot) and the position;
    the action is the new position; the reward, the net P&L."""

    def __init__(self, continua=False, pasos=250):
        self.continua, self.pasos = continua, pasos
        self.observation_space = spaces.Box(-1.0, 1.0, (4,),
                                            np.float32)
        if continua:
            self.action_space = spaces.Box(-1.0, 1.0, (1,),
                                           np.float32)
        else:
            self.action_space = spaces.Discrete(3)

    def _obs(self):
        o = np.zeros(4, np.float32)
        o[self.s], o[3] = 1.0, self.pos
        return o

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        u = self.np_random.random()
        self.s = int(np.searchsorted(estac.cumsum(), u,
                                     side="right"))
        self.pos, self.t = 0.0, 0
        return self._obs(), {}

    def step(self, action):
        if self.continua:
            nueva = float(np.clip(action[0], -1.0, 1.0))
        else:
            nueva = float(POS[int(action)])
        ret = MU[self.s] + SIGMA * self.np_random.standard_normal()
        pnl = nueva * ret - COSTE * abs(nueva - self.pos)
        u = self.np_random.random()
        self.s = int(np.searchsorted(acum[self.s], u, side="right"))
        self.pos, self.t = nueva, self.t + 1
        truncado = self.t >= self.pasos
        return self._obs(), pnl / SIGMA, False, truncado, {"pnl": pnl}

check_env(RegimenEnv(), skip_render_check=True)
check_env(RegimenEnv(continua=True), skip_render_check=True)
print("check_env: no errors in either version")
env = RegimenEnv()
obs, _ = env.reset(seed=1)
print(obs, env.observation_space, env.action_space)

check_env: no errors in either version
[0. 1. 0. 0.] Box(-1.0, 1.0, (4,), float32) Discrete(3)

check_env checks the interface: that reset and step return what they should, that the observations are inside their space and of their type (float32), that the seed makes episodes reproducible. skip_render_check=True skips the test of the render modes, which this environment does not have (without it, it warns that the environment was not created with gymnasium.make). What check_env does not check is the economics: a cost with the wrong sign, the wrong step's return (today's instead of tomorrow's, a leak like those in 5.4.1) or a position that is not stored pass the test all the same. Those are caught by comparing with a known solution, as here, or with no-signal controls.

Policy gradient and PPO

Q-learning learns values and derives the policy from them. Policy gradient methods parametrize the policy directly, $\pi_\theta(a \mid s)$ (a network that gives probabilities for each action), and climb the gradient of the expected reward:

where $\hat{A}_t$ is the advantage: how much better the action $a_t$ turned out than the critic (a second network that estimates the value of the state) expected. Actions that turned out better than expected become more likely. The problem is that a large gradient step can wreck the policy at once, and with noisy rewards that happens often.

PPO (Proximal Policy Optimization, Schulman et al., 2017) limits how much the policy can change in each update. With $\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$, the ratio between the new policy and the one that collected the data, it maximizes

with $\epsilon = 0.2$: beyond a 20% change in the probability of an action, the objective stops rewarding the change. It is the default algorithm in much RL work because of its stability. It works with discrete and continuous actions; here, with the three positions.

The configuration: 8 copies of the environment in parallel (make_vec_env, each with its own seed), 256 steps per copy before each update, a network of two layers of 32 neurons, the same discount as the MDP and seed=0 with device="cpu" and deterministic PyTorch algorithms. 60,000 steps in three blocks, and after each block the agent's deterministic action in the 9 states is read and evaluated with the exact formula.

import torch
from stable_baselines3 import PPO, SAC
from stable_baselines3.common.env_util import make_vec_env

torch.manual_seed(0)
torch.use_deterministic_algorithms(True)

OBS9 = np.zeros((9, 4), np.float32)  # the 9 states
for s in range(3):
    for p in range(3):
        OBS9[3 * s + p, s], OBS9[3 * s + p, 3] = 1.0, POS[p]

def tabla_politica(modelo):
    """The agent's deterministic action in the 9 states."""
    a, _ = modelo.predict(OBS9, deterministic=True)
    return np.asarray(a).reshape(3, 3)

vec = make_vec_env(RegimenEnv, n_envs=8, seed=0)
ppo = PPO("MlpPolicy", vec, gamma=GAMMA, n_steps=256,
          batch_size=256, learning_rate=1e-3,
          policy_kwargs={"net_arch": [32, 32]},
          seed=0, device="cpu", verbose=0)
for bloque in range(3):
    ppo.learn(20_000, reset_num_timesteps=bloque == 0)
    pi_ppo = tabla_politica(ppo)
    print(f"{20_000 * (bloque + 1):>6} steps: "
          f"{int((pi_ppo == pi_opt).sum())}/9 match the optimum, "
          f"{recompensa_media(pi_ppo):.2f} bp/step")
pd.DataFrame(POS[pi_ppo], index=REG,
             columns=["desde corto", "desde fuera", "desde largo"])

 20000 steps: 9/9 match the optimum, 1.90 bp/step
 40000 steps: 8/9 match the optimum, 1.90 bp/step
 60000 steps: 8/9 match the optimum, 1.90 bp/step
         desde corto  desde fuera  desde largo
bajista         -1.0         -1.0         -1.0
neutro          -1.0         -1.0          1.0
alcista          1.0          1.0          1.0

PPO finds the optimal policy by 20,000 steps and then changes the action in one state: in neutral and flat, it opens a short. The long-run mean reward does not move (1.90 bp per step) because that state is not part of the policy's cycle: once in, you are never flat again. It is only visited at the start of each episode, so the agent has very few samples of it and the network decides it by proximity to the neighboring states. It is a general warning: an agent is reliable where it has spent time and arbitrary where it has not, and the rare states of a market (a crisis, a gap) are exactly the ones that matter most.

SAC: continuous actions and entropy

With a continuous position between −1 and 1, the usual choice is SAC (Soft Actor-Critic, Haarnoja et al., 2018): an off-policy method (it reuses old transitions stored in a buffer, which makes it more sample-efficient than PPO) that maximizes the reward plus a bonus for the entropy of the policy:

The entropy term keeps exploration going, and $\alpha$ sets how much. Stable-Baselines3 tunes it automatically (ent_coef="auto") to reach a target entropy. In the tests prior to this lesson, with the same budget of 8,000 steps, the default initial value of $\alpha$ (1.0) left a timid policy, with positions of 0.1 to 0.45 in absolute value where the optimum calls for 1: the entropy weighed more than rewards of a few tenths. With "auto_0.01" (automatic tuning starting at 0.01) it gets much closer. And with the reward divided by 100 (as a percentage) instead of by σ, SAC learned nothing: positions close to zero. They are the two knobs that change the result the most, and neither comes from the financial problem: it is algorithm engineering.

vec_c = make_vec_env(lambda: RegimenEnv(continua=True), n_envs=4,
                     seed=0)
sac = SAC("MlpPolicy", vec_c, gamma=GAMMA, learning_starts=1000,
          batch_size=128, ent_coef="auto_0.01",
          policy_kwargs={"net_arch": [32, 32]},
          seed=0, device="cpu", verbose=0)
sac.learn(8_000)
a_sac = sac.predict(OBS9, deterministic=True)[0].reshape(3, 3)
pd.DataFrame(a_sac, index=REG,
             columns=["desde corto", "desde fuera",
                      "desde largo"]).round(2)

         desde corto  desde fuera  desde largo
bajista        -0.92        -0.65        -0.57
neutro         -0.82         0.42         0.95
alcista         0.87         0.96         0.98

SAC has learned the shape of the optimal policy: long in bullish, short in bearish and, in neutral, a position that grows with the one it already had (the hysteresis). But it does not reach the extremes: in bearish it stays between −0.57 and −0.92 instead of −1. With more steps it would get closer; with 8,000 (4,000 network updates, a few seconds of CPU), this is what it gives. In production it would be trained longer and with several seeds, and even then the gap would have to be measured.

The honest comparison

The exact mean reward only exists for tabular policies. To compare all of them, SAC included, the same 40 episodes of 250 steps (seeds 10,000 to 10,039) are simulated with each policy: same regimes and same returns, so the difference from the optimum does not depend on the randomness of the paths, only on the decisions. The standard error of that difference comes from the 40 episodes.

def evalua(politica, episodios=40, pasos=250):
    """Mean bp/step of each episode, same paths for all."""
    res = []
    for e in range(episodios):
        env = RegimenEnv(continua=True, pasos=pasos)
        o, _ = env.reset(seed=10_000 + e)
        tot = 0.0
        for _ in range(pasos):
            a = np.array([politica(o)], np.float32)
            o, _, _, _, info = env.step(a)
            tot += info["pnl"]
        res.append(tot / pasos)
    return np.array(res)

def de_tabla(pi):
    return lambda o: POS[pi[int(o[:3].argmax()), int(o[3]) + 1]]

def de_sac(o):
    return float(sac.predict(o, deterministic=True)[0][0])

politicas = {"óptima": de_tabla(pi_opt), "Q-learning": de_tabla(pi_q),
             "PPO": de_tabla(pi_ppo), "SAC": de_sac,
             "sin costes": de_tabla(sin_costes),
             "miope": de_tabla(miope)}
res = {k: evalua(f) for k, f in politicas.items()}
filas = []
for k, x in res.items():
    d = x - res["óptima"]
    filas.append({"política": k, "pb/paso": round(x.mean(), 2),
                  "frente al óptimo": round(d.mean(), 2),
                  "error típico": round(d.std(ddof=1) / 40**0.5, 2)})
pd.DataFrame(filas).set_index("política")

            pb/paso  frente al óptimo  error típico
política
óptima         1.99              0.00          0.00
Q-learning     1.99              0.00          0.00
PPO            1.93             -0.06          0.02
SAC            1.55             -0.44          0.13
sin costes     1.28             -0.71          0.15
miope          0.00             -1.99          0.24

The header figure summarizes the table. On these 40 paths the optimum earns 1.99 bp per step (somewhat more than the exact 1.90: it is a sample). Q-learning, with the optimal policy in all 9 states, matches it to the cent. PPO ends up 0.06 bp below, with a standard error of 0.02: it is its mistake in the initial state, which does count in short episodes. SAC ends up 0.44 bp behind (standard error 0.13), 22% below. The rule without costs loses 0.71 and the myopic one, everything.

Three lessons that hold for the whole module. First: without the optimum you would not know that SAC leaves 22% on the table or that PPO fails in one state; in a market there is no optimum, so you have to build references (analytical, as in 5.9.2) and no-signal controls (as in 5.9.3). Second: the algorithms are sensitive to decisions that have nothing financial about them (reward scale, entropy, budget), and a result is only worth as much as its configuration. Third: even in a 9-state problem with a signal twenty times better than a market's, tens of thousands of steps are needed.

Exercises

1. Sweep the cost per unit traded: 6, 12, 18 and 24 bp. For each one, compute the optimal policy, in how many of the 9 states it changes position and its mean reward, alongside that of the rule without costs. When does trading stop paying off?

for c in (6.0, 12.0, 18.0, 24.0):
    COSTE = c  # recompensas() and recompensa_media() read it
    pic = iteracion_valor(recompensas(POS))[0].argmax(axis=2)
    opera = int((POS[pic] != POS[None, :]).sum())
    print(f"cost {c:4.0f} bp: changes in {opera}/9 states, "
          f"optimal {abs(recompensa_media(pic)):5.2f}, "
          f"no costs {recompensa_media(sin_costes):6.2f} bp/step")
COSTE = 6.0  # restored for the rest of the lesson

cost    6 bp: changes in 4/9 states, optimal  1.90, no costs   1.30 bp/step
cost   12 bp: changes in 4/9 states, optimal  1.30, no costs   0.10 bp/step
cost   18 bp: changes in 4/9 states, optimal  0.70, no costs  -1.10 bp/step
cost   24 bp: changes in 0/9 states, optimal  0.00, no costs  -2.30 bp/step

2. Repeat the Q-learning with a constant step $\alpha = 0.05$ instead of $n^{-0.7}$ and compare the matches and the mean reward of its policy at 50,000, 150,000 and 300,000 steps.

Qc = np.zeros((3, 3, 3))
s, p = 1, 1
for k in range(PASOS):
    a = azar[k] if explora[k] else int(Qc[s, p].argmax())
    r = (POS[a] * (MU[s] + SIGMA * ruido[k])
         - COSTE * abs(POS[a] - POS[p]))
    s2 = int(np.searchsorted(acum[s], u[k], side="right"))
    Qc[s, p, a] += 0.05 * (r + GAMMA * Qc[s2, a].max() - Qc[s, p, a])
    s, p = s2, a
    if (k + 1) in (50_000, 150_000, 300_000):
        pi = Qc.argmax(axis=2)
        print(f"{k + 1:>7}: {int((pi == pi_opt).sum())}/9, "
              f"{recompensa_media(pi):.2f} bp/step")

  50000: 4/9, 0.95 bp/step
 150000: 5/9, 0.65 bp/step
 300000: 8/9, 1.81 bp/step

3. A colleague writes the step of their environment with the reward nueva * ret and subtracts the costs afterward, when computing the backtest P&L. Their agent trades at every step. What has happened and what would check_env have detected?