Why reinforcement learning overfits in directional trading.

The worst case for reinforcement learning

The two previous lessons teach RL under its best conditions: an MDP with a known answer (5.9.1) and three control problems with a simulator and an analytical benchmark (5.9.2). Directional trading, deciding every day whether to be long, short or out of an asset, is the opposite, and it is what gets published most. Four reasons make it the worst case:

• A single history. The agent in 5.9.2 is trained on thousands of simulated paths. Here there is only one path, the historical one, and the dice cannot be rolled again. Ten years of sessions are 2,500 transitions, and the training “epochs” go over them again and again.
• Tiny signal-to-noise ratio. An IC of a few hundredths (lesson 5.1.1) means that almost all of the reward at each step is noise. In the MDP of 5.9.1 the signal was a quarter of the per-step deviation and Q-learning needed 140,000 steps.
• Non-stationarity. The market changes (lesson 5.10.2): what is learned from 2008-2017 need not hold in 2018-2025.
• Very flexible policies. A network with thousands of parameters and a decision at every step can fit any sequence, and the hyperparameters (network, learning rate, steps, seed) multiply the number of tests.

flowchart TB
  A[A single history] --> E[Flexible agent]
  B[Minimal signal] --> E
  C[Changing market] --> E
  E --> F[Learns the noise]
  F --> G[Brilliant in sample]
  F --> H[Nothing out of sample]

The lesson measures this with data where we know what there is to learn: a series with nothing to learn and its twin with a small planted edge.

A control and a planted edge

The control is the divisa asset in vol5_diario.parquet: a synthetic exchange rate whose conditional mean is constant and equal to zero (μ = 0 in the README table), with stochastic volatility but no predictability at all in the returns. The README checks it: an AR(5) gives an out-of-sample R² of −0.15%. Anything an agent “earns” here, out of sample, is noise.

The twin series adds a small edge with a known answer. A signal $s_t$, an AR(1) with persistence 0.9 and variance 1, predicts the next day's return:

with $\sigma$ the daily standard deviation of the currency over the training period. The true IC is 0.10, generous for a real market and small enough for the noise of the sample to matter. The agent observes the same thing in both series: the last 10 standardized returns, the signal $s_t$ and its position. In the control, $s_t$ is noise unrelated to anything. Training: 2008-2017; test: 2018-2025.

import zlib
import numpy as np
import pandas as pd

v = pd.read_parquet("vol5_diario.parquet")
div = v[v["activo"] == "divisa"].set_index("fecha")["rendimiento"]
r_ctrl, fechas = div.to_numpy(), div.index
corte = int((fechas < "2018-01-01").sum())
SD = r_ctrl[:corte].std()

rng = np.random.default_rng([2029, zlib.crc32(b"n5_l593_senal")])
e = rng.standard_normal(len(r_ctrl))
s = np.zeros(len(r_ctrl))
for t in range(1, len(s)):
    s[t] = 0.9 * s[t - 1] + np.sqrt(1 - 0.9**2) * e[t]
KAPPA = 0.10
r_edge = r_ctrl.copy()
r_edge[1:] += KAPPA * SD * s[:-1]
SERIES = {"control": r_ctrl, "edge": r_edge}

def ic(r, a, b):
    return np.corrcoef(s[a:b - 1], r[a + 1:b])[0, 1]

print(f"{corte} training sessions, {len(s) - corte} test "
      f"sessions; daily deviation {SD:.2%}")
for k, r in SERIES.items():
    print(f"{k:<8} signal IC: training "
          f"{ic(r, 0, corte):+.3f}, test {ic(r, corte, len(r)):+.3f}")

2518 training sessions, 2011 test sessions; daily deviation 0.56%
control  signal IC: training +0.025, test -0.019
edge     signal IC: training +0.122, test +0.077

The sample already shows the problem before anything is trained. In the control, the signal has no relation to the returns, and yet its measured IC is +0.025 in training and −0.019 in the test: with 2,500 days, the standard error of an IC is 1/√2,500 = 0.02, so anything between −0.04 and +0.04 is compatible with zero. In the series with an edge, the true IC is 0.10, but the measured one is 0.122 in training (the sample was lucky) and 0.077 in the test (less lucky). An agent trained on 2008-2017 sees an edge 22% larger than the real one; one evaluated on 2018-2025, one 23% smaller. Both are the same edge.

The directional environment

The environment follows the rules of 5.9.1: position in the observation, costs inside the reward, reward of order 1. Each episode is a window of 250 sessions chosen at random within the training period (otherwise the agent would always see the same sequence in the same order). The cost is 5 bp per unit traded, a little under a tenth of the daily deviation. The coste_solo_al_abrir parameter reproduces a programming bug used further below.

import gymnasium as gym
from gymnasium import spaces
from gymnasium.utils.env_checker import check_env

L, COSTE = 10, 0.0005  # 10 lagged returns; 5 bp

class DireccionalEnv(gym.Env):
    def __init__(self, r, ini=L, fin=corte, ventana=250,
                 coste_solo_al_abrir=False):
        self.r, self.ini, self.fin = r, ini, fin
        self.ventana, self.solo_abrir = ventana, coste_solo_al_abrir
        self.observation_space = spaces.Box(-10.0, 10.0, (L + 2,),
                                            np.float32)
        self.action_space = spaces.Discrete(3)

    def _obs(self):
        t = self.t
        lags = np.clip(self.r[t - L + 1:t + 1] / SD, -10, 10)
        return np.r_[lags, s[t], self.pos].astype(np.float32)

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        hi = self.fin - self.ventana
        self.t = int(self.np_random.integers(self.ini, hi))
        self.t_fin, self.pos = self.t + self.ventana, 0.0
        return self._obs(), {}

    def step(self, a):
        nueva = float(a) - 1.0
        if self.solo_abrir:  # bug: flipping and closing are free
            coste = COSTE * abs(nueva) if self.pos == 0 else 0.0
        else:
            coste = COSTE * abs(nueva - self.pos)
        pnl = nueva * self.r[self.t + 1] - coste
        self.pos, self.t = nueva, self.t + 1
        return self._obs(), pnl / SD, False, self.t >= self.t_fin, {}

check_env(DireccionalEnv(r_ctrl), skip_render_check=True)
print("check_env: no errors")

check_env: no errors

The evaluation does not use the environment: it runs through the whole period, in one go, with the agent's deterministic policy and computes the P&L with the correct costs, whatever reward it was trained with. Since the decision depends on the current position, which can only be −1, 0 or 1, the agent is asked for its action for the three possible positions on all dates at once, and the position is then chained.

def evalua(politica, r, ini, fin):
    """Annual Sharpe and daily turnover of a policy on [ini, fin)
    with the real cost, chaining the position from 0."""
    t = np.arange(max(ini, L), fin - 1)
    lags = np.clip(np.stack([r[t - L + 1 + j] for j in range(L)],
                            axis=1) / SD, -10, 10)
    acc = np.array([politica(np.column_stack(
        [lags, s[t], np.full(len(t), p)]).astype(np.float32))
        for p in (-1.0, 0.0, 1.0)])
    pos, pnl, rot = 0.0, np.empty(len(t)), 0.0
    for i, tt in enumerate(t):
        nueva = acc[int(pos) + 1, i]
        pnl[i] = nueva * r[tt + 1] - COSTE * abs(nueva - pos)
        rot += abs(nueva - pos)
        pos = nueva
    sharpe = pnl.mean() / pnl.std() * np.sqrt(252) if pnl.std() else 0.0
    return round(sharpe, 2), round(rot / len(t), 2)

def de_agente(m):
    return lambda o: m.predict(o, deterministic=True)[0] - 1.0

largo = lambda o: np.ones(len(o))
oraculo = lambda o: np.sign(o[:, L])  # sign of the true signal
filas = []
for k, r in SERIES.items():
    for nombre, f in (("buy and hold", largo),
                      ("oracle: sign of s", oraculo)):
        filas.append([k, nombre, *evalua(f, r, 0, corte),
                      *evalua(f, r, corte, len(r))])
ref = pd.DataFrame(filas, columns=["series", "rule", "Sharpe in",
                                   "turnover in", "Sharpe out",
                                   "turnover out"])
ref

    series               rule  Sharpe in  turnover in  Sharpe out  turnover out
0  control       buy and hold       0.60         0.00       -0.18          0.00
1  control  oracle: sign of s      -0.13         0.34       -0.65          0.26
2     edge       buy and hold       0.57         0.00       -0.13          0.00
3     edge  oracle: sign of s       1.08         0.34        0.58          0.26

The two fixed rules provide the yardstick. Buying and holding the currency earns a Sharpe of 0.60 in 2008-2017 and loses −0.18 in 2018-2025, with a true mean of zero: the drift of the training sample is an accident of about two standard errors, and any agent that learns to be long will inherit it. The oracle (position = sign of the true signal, switching every time the sign changes) gives 1.08 in sample and 0.58 out of sample on the series with an edge, with a turnover of 0.26 to 0.34 units per day; on the control it loses, because it pays costs to follow noise. The naive oracle is not the best possible policy with costs: as in 5.9.1, trading less (a no-trade zone) improves the net result.

In and out of sample, with three seeds

Each agent is a PPO with a two-layer network of 64 neurons, 8 parallel environments and a discount of 0.9 (a horizon of about 10 days: the signal has a half-life of 6.6), trained for 60,000 steps (about 24 passes over the 2,500 training days) in two blocks. After each block it is evaluated in and out of sample. Three seeds per series: the seed changes the network initialization, the episode windows and the exploration.

import torch
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env

torch.use_deterministic_algorithms(True)

def entrena(r, semilla, bloques=2, **kw_env):
    torch.manual_seed(semilla)
    vec = make_vec_env(lambda: DireccionalEnv(r, **kw_env), n_envs=8,
                       seed=semilla)
    m = PPO("MlpPolicy", vec, gamma=0.9, n_steps=250, batch_size=500,
            n_epochs=5, learning_rate=3e-4,
            policy_kwargs={"net_arch": [64, 64]},
            seed=semilla, device="cpu", verbose=0)
    hist = []
    for b in range(bloques):
        m.learn(30_000, reset_num_timesteps=b == 0)
        f = de_agente(m)
        hist.append([30_000 * (b + 1), *evalua(f, r, 0, corte),
                     *evalua(f, r, corte, len(r))])
    return m, hist

filas, agentes = [], {}
for k, r in SERIES.items():
    for semilla in range(3):
        m, hist = entrena(r, semilla)
        agentes[(k, semilla)] = m
        filas += [[k, semilla, *h] for h in hist]
res = pd.DataFrame(filas, columns=["series", "seed", "steps",
                                   "Sharpe in", "turn. in",
                                   "Sharpe out", "turn. out"])
res

     series  seed  steps  Sharpe in  turn. in  Sharpe out  turn. out
0   control     0  30000       0.65      0.22       -1.17       0.22
1   control     0  60000       0.56      0.02       -0.19       0.02
2   control     1  30000       0.34      0.18       -0.74       0.17
3   control     1  60000       0.73      0.08       -0.53       0.07
4   control     2  30000       0.34      0.21       -0.65       0.21
5   control     2  60000       0.45      0.02        0.03       0.01
6      edge     0  30000       1.60      0.14        0.38       0.15
7      edge     0  60000       1.84      0.11        0.63       0.11
8      edge     1  30000       1.46      0.14        0.49       0.13
9      edge     1  60000       1.98      0.10        0.80       0.10
10     edge     2  30000       1.42      0.11        0.52       0.10
11     edge     2  60000       1.77      0.08        0.81       0.08

The figure at the top puts the twelve points on the in-sample versus out-of-sample plane. Three readings:

• Control. At 60,000 steps, the three agents have an in-sample Sharpe of 0.45 to 0.73 and barely trade (turnover of 0.02 to 0.08): they have learned to be long, the chance drift of 2008-2017. Out of sample, −0.53 to +0.03. At 30,000 steps they were still trading (turnover of 0.2 per day) following patterns in the noise, and out of sample they lost between −0.65 and −1.17. No control agent has learned anything, and in sample they all seemed to win.
• Edge. The agents pick up the planted edge: out of sample, 0.63 to 0.81, above the naive oracle (0.58) because they learn to trade less (turnover of 0.08 to 0.11 against 0.26). It is the only good thing in the table, and it is possible because there is something to learn and the costs are set correctly in the reward.
• The gap. In sample, the agents on the series with an edge promise 1.77 to 1.98: more than twice what they deliver out of sample. Training more raises the in-sample Sharpe for all three seeds (from 30,000 to 60,000 steps); the out-of-sample one also rises in this configuration, but exercise 1 shows that this does not last. With real data, only the in-sample column would be seen if validation were done badly, and the difference between a Sharpe of 1.9 and 0.7 is the difference between an exceptional strategy and an ordinary one.

The seed is also a hyperparameter

Six agents, six results. With real data nobody knows what the oracle is, and the temptation is to present the agent with the best in-sample Sharpe or, worse, the one with the best out-of-sample Sharpe after having seen all six. Both are multiple tests (lesson 2.5.4). The standard error of an annual Sharpe measured over $T$ years is about $\sqrt{(1 + SR^2/2)/T}$, and the maximum of $N$ Sharpes with no edge at all has a positive expectation:

from scipy.stats import norm

def error_sharpe(sr, anos):
    return np.sqrt((1 + sr**2 / 2) / anos)

def maximo_esperado(n, error):
    """Expected maximum of n Sharpes with mean 0 (Bailey and López de
    Prado; lesson 2.5.4)."""
    g = 0.5772156649  # Euler-Mascheroni constant
    return error * ((1 - g) * norm.ppf(1 - 1 / n)
                    + g * norm.ppf(1 - 1 / (n * np.e)))

anos_dentro, anos_fuera = corte / 252, (len(s) - corte) / 252
print(f"standard error of the Sharpe: {error_sharpe(0, anos_dentro):.2f}"
      f" over {anos_dentro:.0f} years, {error_sharpe(0, anos_fuera):.2f}"
      f" over {anos_fuera:.0f} years")
for n in (3, 6, 20):
    print(f"expected maximum of {n:>2} Sharpes with no edge (out): "
          f"{maximo_esperado(n, error_sharpe(0, anos_fuera)):.2f}")
fin = res[res["steps"] == 60_000].set_index(["series", "seed"])
for k in SERIES:
    x = fin.loc[k]
    mejor = x["Sharpe in"].idxmax()
    print(f"{k:<8} out: mean {x['Sharpe out'].mean():+.2f}, "
          f"range {x['Sharpe out'].min():+.2f} to "
          f"{x['Sharpe out'].max():+.2f}; the best in sample "
          f"(seed {mejor}) gives {x.loc[mejor, 'Sharpe out']:+.2f}")

standard error of the Sharpe: 0.32 over 10 years, 0.35 over 8 years
expected maximum of  3 Sharpes with no edge (out): 0.30
expected maximum of  6 Sharpes with no edge (out): 0.46
expected maximum of 20 Sharpes with no edge (out): 0.67
control  out: mean -0.23, range -0.53 to +0.03; the best in sample (seed 1) gives -0.53
edge     out: mean +0.75, range +0.63 to +0.81; the best in sample (seed 1) gives +0.80

With a standard error of 0.35 over 8 years, the dispersion across seeds on the series with an edge (0.63 to 0.81) is small compared with the error of the sample itself: the three seeds see the same 2,011 test days, and if those days had been different, the three figures would move together. On the control, the dispersion is larger (−0.53 to +0.03) and choosing by the in-sample Sharpe leads to the worst out-of-sample seed (seed 1). The table of expected maxima gives the scale: with 6 agents with no edge at all, the expected best out-of-sample Sharpe is already 0.46, and with 20 configurations (a few architectures, learning rates and seeds, something any project reaches in an afternoon), 0.67. An out-of-sample 0.7 chosen among twenty attempts proves nothing; here it proves something only because the control is known. The Deflated Sharpe of lesson 2.5.4 formalizes the correction; Henderson et al. (2018) document the same seed problem in RL papers in general, outside finance.

Reward hacking: a bug in the costs

The last experiment is a very common programming bug. Whoever writes the environment charges the cost “when opening a position”: if you are flat and enter, you pay; if you are already in, you do not. It seems reasonable until you think about the flip: going from long to short, which in reality costs twice as much as opening, is free, and so is closing. It is the coste_solo_al_abrir=True variant of the environment. An agent is trained with it on the series with an edge (seed 0) and evaluated, as always, with the correct costs.

m_hack, hist_hack = entrena(SERIES["edge"], 0,
                            coste_solo_al_abrir=True)
honesto = res[(res["series"] == "edge") & (res["seed"] == 0)]
comp = pd.DataFrame(
    [["correct cost", *honesto.iloc[-1, 3:]],
     ["cost only on opening", *hist_hack[-1][1:]]],
    columns=["training reward", "Sharpe in",
             "turn. in", "Sharpe out", "turn. out"])

def sharpe_segun_recompensa(m, r):
    """In-sample Sharpe with the costs of the hacked environment:
    what the agent believes it earns."""
    env = DireccionalEnv(r, ventana=corte - L - 1,
                         coste_solo_al_abrir=True)
    o, _ = env.reset(seed=0)
    env.t, env.t_fin, env.pos = L, corte - 1, 0.0
    o, pnl, fin = env._obs(), [], False
    while not fin:
        a = m.predict(o, deterministic=True)[0]
        o, rec, _, fin, _ = env.step(a)
        pnl.append(rec * SD)
    pnl = np.array(pnl)
    return round(pnl.mean() / pnl.std() * np.sqrt(252), 2)

print("In-sample Sharpe according to the hacked reward:",
      sharpe_segun_recompensa(m_hack, SERIES["edge"]))
comp

In-sample Sharpe according to the hacked reward: 2.03
        training reward  Sharpe in  turn. in  Sharpe out  turn. out
0          correct cost       1.84      0.11        0.63       0.11
1  cost only on opening       1.48      0.40        0.37       0.34

The agent with the hacked reward believes it earns an in-sample Sharpe of 2.03, more than the honest one. With the real costs, 1.48 in sample and 0.37 out of sample, against 1.84 and 0.63 for the agent with the cost programmed correctly. The figure compares the two. The cause is the turnover: 0.40 units per day in sample, almost four times that of the honest agent (0.11). For it, flipping from long to short and closing cost nothing, so it follows every change in the signal and in the noise of the lags; the evaluation with the real costs charges it for all those flips. This is reward hacking (or specification gaming): the agent optimizes the reward it was given, not the one it was meant to be given, and it is better than a person at finding the loophole.

Two defenses, both cheap. Always evaluate with a P&L calculation independent of the training environment (like evalua in this lesson), and watch the turnover: an agent that turns over much more than the oracle, or than a simple rule with the same information, is exploiting something. A unit test of the environment with known flips (from +1 to −1 must cost twice the unit cost) would have caught this bug before training; check_env would not, because it checks the interface, not the accounting.

FinRL as a critical reference

FinRL is the best-known RL library for trading: it is maintained by the AI4Finance Foundation, described in a 2020 paper (Liu et al., arXiv:2011.09607) and offers environments with the Gymnasium interface for stocks, crypto and portfolios, agents from Stable-Baselines3, ElegantRL and RLlib, data downloaders and end-to-end tutorials. What was checked in its sources as of October 23, 2026:

The PyPI package is two and a half years behind the repository: pip install finrl installs code from 2024. It is not installed in the course environment; the typical flow of its tutorials, with its current API as read in the repository, is this (it needs network access to download prices from Yahoo):

from finrl.agents.stablebaselines3.models import DRLAgent
from finrl.config import INDICATORS
from finrl.meta.env_stock_trading.env_stocktrading import (
    StockTradingEnv)
from finrl.meta.preprocessor.preprocessors import (FeatureEngineer,
                                                   data_split)
from finrl.meta.preprocessor.yahoodownloader import YahooDownloader

df = YahooDownloader(start_date="2009-01-01", end_date="2024-01-01",
                     ticker_list=["AAPL", "MSFT", "JPM"]).fetch_data()
df = FeatureEngineer(use_technical_indicator=True,
                     tech_indicator_list=INDICATORS,
                     use_turbulence=True).preprocess_data(df)
train = data_split(df, "2009-01-01", "2020-07-01")
trade = data_split(df, "2020-07-01", "2024-01-01")

n = train["tic"].nunique()
kw = dict(stock_dim=n, hmax=100, initial_amount=1_000_000,
          num_stock_shares=[0] * n, buy_cost_pct=[0.001] * n,
          sell_cost_pct=[0.001] * n, reward_scaling=1e-4,
          state_space=1 + 2 * n + len(INDICATORS) * n,
          action_space=n, tech_indicator_list=INDICATORS)
env_train, _ = StockTradingEnv(df=train, **kw).get_sb_env()
agente = DRLAgent(env=env_train)
ppo = agente.get_model("ppo")
ppo = DRLAgent.train_model(ppo, tb_log_name="ppo",
                           total_timesteps=50_000)
cuenta, acciones = DRLAgent.DRL_prediction(
    model=ppo, environment=StockTradingEnv(df=trade, **kw))

None of this is wrong in itself: it is a convenient way to set up the experiment. The problem is what it does not do, and a tutorial result being read as evidence:

• A single training-test split, with no walk-forward (lesson 5.4.3) or purging; if the hyperparameters or the number of steps are tuned by looking at the test period, the test stops being one.
• One seed by default (seed=None in get_model): the result changes on every run, and without several seeds you do not know by how much (this module: several tenths of Sharpe).
• A universe of current components: in October 2026 the repository itself merged pull request 1462, which clarifies the survivorship bias of a tutorial's Dow backtest and switches to point-in-time components. Results published earlier with that tutorial carry the bias inside them (lessons 1.4.4 and 2.5.1).
• No control: no tutorial compares with the same agent trained on a series with no signal, which is what separates learning an edge from learning the sample.
• A comparison with buy and hold or with an index over a specific period, with no standard error: with 3 years of test data, the standard error of the Sharpe is almost 0.6.

Before believing a directional agent

1. Is there a control? The same agent, with the same configuration, trained on a series with no signal (permuted or synthetic) has to give out of sample what is expected from chance.
2. Truly out of sample? A test after the training and after the hyperparameter choice, ideally in walk-forward with retraining (lesson 5.4.3).
3. How many seeds and configurations? All of them, with their dispersion, and the Sharpe deflated by the number of tests (lesson 2.5.4).
4. Full net P&L? Computed outside the training environment, with real costs, slippage and financing, and the turnover in plain view.
5. Against what? Buy and hold, the simplest linear rule with the same information (5.5.1) and, if one exists, an analytical benchmark. An agent that does not beat a regression with the same features has learned nothing worth its complexity.
6. Point-in-time data? A universe with the companies that existed on each date and features with the date on which they were known (lesson 5.2.2).

Exercises

1. Repeat the training on the series with an edge, seed 0, with twice as many steps (four blocks of 30,000). Does the in-sample Sharpe go up? And the out-of-sample one?

_, hist_largo = entrena(SERIES["edge"], 0, bloques=4)
pd.DataFrame(hist_largo, columns=["steps", "Sharpe in",
                                  "turn. in", "Sharpe out",
                                  "turn. out"])

    steps  Sharpe in  turn. in  Sharpe out  turn. out
0   30000       1.60      0.14        0.38       0.15
1   60000       1.84      0.11        0.63       0.11
2   90000       2.10      0.11        0.71       0.12
3  120000       2.06      0.15        0.55       0.16

2. Using the standard error of the Sharpe, how many years of data would be needed for the out-of-sample Sharpe of the oracle on the series with an edge (the expected one, not the realized one) to be two standard errors above zero?

# Expected annual Sharpe of the gross oracle: kappa · E|s| · sqrt(252)
sr = KAPPA * np.sqrt(2 / np.pi) * np.sqrt(252)
neto = sr - COSTE * 0.2 / SD * np.sqrt(252)  # turnover ≈ 0.2/day
for nombre, x in (("gross", sr), ("net-of-cost", neto)):
    anos = 4 * (1 + x**2 / 2) / x**2
    print(f"Expected {nombre} Sharpe: {x:.2f}; years for 2 standard "
          f"errors: {anos:.0f}")

Expected gross Sharpe: 1.27; years for 2 standard errors: 4
Expected net-of-cost Sharpe: 0.98; years for 2 standard errors: 6

3. A paper presents an RL agent that, over five years of testing on 30 stocks, obtains a Sharpe of 2.1 against 0.8 for buy and hold. It does not say how many configurations or how many seeds it tried. Which three pieces of information would you ask for, and why?