--- title: "5.9.3 Why RL overfits in directional trading" description: "A PPO agent on a series with no predictability and on one with a small edge: in and out of sample, seeds, hacked rewards and FinRL." url: "https://pythonparatrading.com/en/python-trading-course/level-5/why-rl-overfits-directional-trading/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-5/why-rl-overfits-directional-trading.png" --- # 5.9.3 · Why reinforcement learning overfits in directional trading > Training an agent to buy and sell an asset is the most published application of reinforcement learning and the one that holds up worst under honest validation. This lesson does it with a control that has no predictability at all and with a twin series carrying a small planted edge, and measures what matters: the gap between in-sample and out-of-sample, how much the result changes with the seed and what an agent learns when the reward has a bug in the costs. - Course: [Level 5: Machine learning and deep learning (Keras/PyTorch)](https://pythonparatrading.com/en/python-trading-course/level-5/) - Module: 5.9 · Reinforcement learning - Duration: 65 min - Prerequisites: [5.9.1 · MDP, Q-learning, PPO and Gymnasium environments](https://pythonparatrading.com/en/python-trading-course/level-5/mdp-q-learning-ppo-gymnasium/), [5.9.2 · RL for execution, market making and dynamic hedging](https://pythonparatrading.com/en/python-trading-course/level-5/rl-execution-market-making-hedging/), [2.5.4 · Deflated Sharpe Ratio and Probability of Backtest Overfitting (PBO)](https://pythonparatrading.com/en/python-trading-course/level-2/deflated-sharpe-ratio-and-pbo/), [5.4.3 · Walk-forward with retraining: windows and frequency](https://pythonparatrading.com/en/python-trading-course/level-5/walk-forward-retraining-windows/) ## By the end of this lesson - Explain why directional trading is the worst case for RL: a single history, a very low signal-to-noise ratio, non-stationarity and very flexible policies. - Train a PPO agent on a control with no predictability and on a series with a planted edge, and compare in-sample and out-of-sample results with the oracle and with buy and hold. - Measure the variance across seeds and treat seeds as multiple tests (Deflated Sharpe). - Recognize reward hacking: an agent that exploits a bug in the costs of the reward, and why evaluation must always use the full net P&L. - Read FinRL critically: what it is, its license and maintenance, and what has to be checked before believing a result produced with it. ## The worst case for reinforcement learning The two previous lessons teach RL under its best conditions: an MDP with a known answer (5.9.1) and three control problems with a simulator and an analytical benchmark (5.9.2). Directional trading, deciding every day whether to be long, short or out of an asset, is the opposite, and it is what gets published most. Four reasons make it the worst case: - **A single history.** The agent in 5.9.2 is trained on thousands of simulated paths. Here there is only one path, the historical one, and the dice cannot be rolled again. Ten years of sessions are 2,500 transitions, and the training “epochs” go over them again and again. - **Tiny signal-to-noise ratio.** An IC of a few hundredths (lesson 5.1.1) means that almost all of the reward at each step is noise. In the MDP of 5.9.1 the signal was a quarter of the per-step deviation and Q-learning needed 140,000 steps. - **Non-stationarity.** The market changes (lesson 5.10.2): what is learned from 2008-2017 need not hold in 2018-2025. - **Very flexible policies.** A network with thousands of parameters and a decision at every step can fit any sequence, and the hyperparameters (network, learning rate, steps, seed) multiply the number of tests. ```mermaid flowchart TB A[A single history] --> E[Flexible agent] B[Minimal signal] --> E C[Changing market] --> E E --> F[Learns the noise] F --> G[Brilliant in sample] F --> H[Nothing out of sample] ``` The lesson measures this with data where we know what there is to learn: a series with nothing to learn and its twin with a small planted edge. ## A control and a planted edge The control is the `divisa` asset in `vol5_diario.parquet`: a synthetic exchange rate whose conditional mean is constant and equal to zero (μ = 0 in the README table), with stochastic volatility but no predictability at all in the returns. The README checks it: an AR(5) gives an out-of-sample R² of −0.15%. Anything an agent “earns” here, out of sample, is noise. The twin series adds a small edge with a known answer. A signal $s_t$, an AR(1) with persistence 0.9 and variance 1, predicts the next day's return: $$ r^{B}_{t+1} = r_{t+1} + \kappa\,\sigma\,s_t, \qquad \kappa = 0.10 $$ with $\sigma$ the daily standard deviation of the currency over the training period. The true IC is 0.10, generous for a real market and small enough for the noise of the sample to matter. The agent observes the same thing in both series: the last 10 standardized returns, the signal $s_t$ and its position. In the control, $s_t$ is noise unrelated to anything. Training: 2008-2017; test: 2018-2025. ```python import zlib import numpy as np import pandas as pd v = pd.read_parquet("vol5_diario.parquet") div = v[v["activo"] == "divisa"].set_index("fecha")["rendimiento"] r_ctrl, fechas = div.to_numpy(), div.index corte = int((fechas < "2018-01-01").sum()) SD = r_ctrl[:corte].std() rng = np.random.default_rng([2029, zlib.crc32(b"n5_l593_senal")]) e = rng.standard_normal(len(r_ctrl)) s = np.zeros(len(r_ctrl)) for t in range(1, len(s)): s[t] = 0.9 * s[t - 1] + np.sqrt(1 - 0.9**2) * e[t] KAPPA = 0.10 r_edge = r_ctrl.copy() r_edge[1:] += KAPPA * SD * s[:-1] SERIES = {"control": r_ctrl, "edge": r_edge} def ic(r, a, b): return np.corrcoef(s[a:b - 1], r[a + 1:b])[0, 1] print(f"{corte} training sessions, {len(s) - corte} test " f"sessions; daily deviation {SD:.2%}") for k, r in SERIES.items(): print(f"{k:<8} signal IC: training " f"{ic(r, 0, corte):+.3f}, test {ic(r, corte, len(r)):+.3f}") ``` ```output 2518 training sessions, 2011 test sessions; daily deviation 0.56% control signal IC: training +0.025, test -0.019 edge signal IC: training +0.122, test +0.077 ``` The sample already shows the problem before anything is trained. In the control, the signal has no relation to the returns, and yet its measured IC is +0.025 in training and −0.019 in the test: with 2,500 days, the standard error of an IC is 1/√2,500 = 0.02, so anything between −0.04 and +0.04 is compatible with zero. In the series with an edge, the true IC is 0.10, but the measured one is 0.122 in training (the sample was lucky) and 0.077 in the test (less lucky). An agent trained on 2008-2017 sees an edge 22% larger than the real one; one evaluated on 2018-2025, one 23% smaller. Both are the same edge. ## The directional environment The environment follows the rules of 5.9.1: position in the observation, costs inside the reward, reward of order 1. Each episode is a window of 250 sessions chosen at random within the training period (otherwise the agent would always see the same sequence in the same order). The cost is 5 bp per unit traded, a little under a tenth of the daily deviation. The `coste_solo_al_abrir` parameter reproduces a programming bug used further below. ```python import gymnasium as gym from gymnasium import spaces from gymnasium.utils.env_checker import check_env L, COSTE = 10, 0.0005 # 10 lagged returns; 5 bp class DireccionalEnv(gym.Env): def __init__(self, r, ini=L, fin=corte, ventana=250, coste_solo_al_abrir=False): self.r, self.ini, self.fin = r, ini, fin self.ventana, self.solo_abrir = ventana, coste_solo_al_abrir self.observation_space = spaces.Box(-10.0, 10.0, (L + 2,), np.float32) self.action_space = spaces.Discrete(3) def _obs(self): t = self.t lags = np.clip(self.r[t - L + 1:t + 1] / SD, -10, 10) return np.r_[lags, s[t], self.pos].astype(np.float32) def reset(self, seed=None, options=None): super().reset(seed=seed) hi = self.fin - self.ventana self.t = int(self.np_random.integers(self.ini, hi)) self.t_fin, self.pos = self.t + self.ventana, 0.0 return self._obs(), {} def step(self, a): nueva = float(a) - 1.0 if self.solo_abrir: # bug: flipping and closing are free coste = COSTE * abs(nueva) if self.pos == 0 else 0.0 else: coste = COSTE * abs(nueva - self.pos) pnl = nueva * self.r[self.t + 1] - coste self.pos, self.t = nueva, self.t + 1 return self._obs(), pnl / SD, False, self.t >= self.t_fin, {} check_env(DireccionalEnv(r_ctrl), skip_render_check=True) print("check_env: no errors") ``` ```output check_env: no errors ``` The evaluation does not use the environment: it runs through the whole period, in one go, with the agent's deterministic policy and computes the P&L with the correct costs, whatever reward it was trained with. Since the decision depends on the current position, which can only be −1, 0 or 1, the agent is asked for its action for the three possible positions on all dates at once, and the position is then chained. ```python def evalua(politica, r, ini, fin): """Annual Sharpe and daily turnover of a policy on [ini, fin) with the real cost, chaining the position from 0.""" t = np.arange(max(ini, L), fin - 1) lags = np.clip(np.stack([r[t - L + 1 + j] for j in range(L)], axis=1) / SD, -10, 10) acc = np.array([politica(np.column_stack( [lags, s[t], np.full(len(t), p)]).astype(np.float32)) for p in (-1.0, 0.0, 1.0)]) pos, pnl, rot = 0.0, np.empty(len(t)), 0.0 for i, tt in enumerate(t): nueva = acc[int(pos) + 1, i] pnl[i] = nueva * r[tt + 1] - COSTE * abs(nueva - pos) rot += abs(nueva - pos) pos = nueva sharpe = pnl.mean() / pnl.std() * np.sqrt(252) if pnl.std() else 0.0 return round(sharpe, 2), round(rot / len(t), 2) def de_agente(m): return lambda o: m.predict(o, deterministic=True)[0] - 1.0 largo = lambda o: np.ones(len(o)) oraculo = lambda o: np.sign(o[:, L]) # sign of the true signal filas = [] for k, r in SERIES.items(): for nombre, f in (("buy and hold", largo), ("oracle: sign of s", oraculo)): filas.append([k, nombre, *evalua(f, r, 0, corte), *evalua(f, r, corte, len(r))]) ref = pd.DataFrame(filas, columns=["series", "rule", "Sharpe in", "turnover in", "Sharpe out", "turnover out"]) ref ``` ```output series rule Sharpe in turnover in Sharpe out turnover out 0 control buy and hold 0.60 0.00 -0.18 0.00 1 control oracle: sign of s -0.13 0.34 -0.65 0.26 2 edge buy and hold 0.57 0.00 -0.13 0.00 3 edge oracle: sign of s 1.08 0.34 0.58 0.26 ``` The two fixed rules provide the yardstick. Buying and holding the currency earns a Sharpe of 0.60 in 2008-2017 and loses −0.18 in 2018-2025, with a true mean of zero: the drift of the training sample is an accident of about two standard errors, and any agent that learns to be long will inherit it. The oracle (position = sign of the true signal, switching every time the sign changes) gives 1.08 in sample and 0.58 out of sample on the series with an edge, with a turnover of 0.26 to 0.34 units per day; on the control it loses, because it pays costs to follow noise. The naive oracle is not the best possible policy with costs: as in 5.9.1, trading less (a no-trade zone) improves the net result. ## In and out of sample, with three seeds Each agent is a PPO with a two-layer network of 64 neurons, 8 parallel environments and a discount of 0.9 (a horizon of about 10 days: the signal has a half-life of 6.6), trained for 60,000 steps (about 24 passes over the 2,500 training days) in two blocks. After each block it is evaluated in and out of sample. Three seeds per series: the seed changes the network initialization, the episode windows and the exploration. ```python import torch from stable_baselines3 import PPO from stable_baselines3.common.env_util import make_vec_env torch.use_deterministic_algorithms(True) def entrena(r, semilla, bloques=2, **kw_env): torch.manual_seed(semilla) vec = make_vec_env(lambda: DireccionalEnv(r, **kw_env), n_envs=8, seed=semilla) m = PPO("MlpPolicy", vec, gamma=0.9, n_steps=250, batch_size=500, n_epochs=5, learning_rate=3e-4, policy_kwargs={"net_arch": [64, 64]}, seed=semilla, device="cpu", verbose=0) hist = [] for b in range(bloques): m.learn(30_000, reset_num_timesteps=b == 0) f = de_agente(m) hist.append([30_000 * (b + 1), *evalua(f, r, 0, corte), *evalua(f, r, corte, len(r))]) return m, hist filas, agentes = [], {} for k, r in SERIES.items(): for semilla in range(3): m, hist = entrena(r, semilla) agentes[(k, semilla)] = m filas += [[k, semilla, *h] for h in hist] res = pd.DataFrame(filas, columns=["series", "seed", "steps", "Sharpe in", "turn. in", "Sharpe out", "turn. out"]) res ``` ```output series seed steps Sharpe in turn. in Sharpe out turn. out 0 control 0 30000 0.65 0.22 -1.17 0.22 1 control 0 60000 0.56 0.02 -0.19 0.02 2 control 1 30000 0.34 0.18 -0.74 0.17 3 control 1 60000 0.73 0.08 -0.53 0.07 4 control 2 30000 0.34 0.21 -0.65 0.21 5 control 2 60000 0.45 0.02 0.03 0.01 6 edge 0 30000 1.60 0.14 0.38 0.15 7 edge 0 60000 1.84 0.11 0.63 0.11 8 edge 1 30000 1.46 0.14 0.49 0.13 9 edge 1 60000 1.98 0.10 0.80 0.10 10 edge 2 30000 1.42 0.11 0.52 0.10 11 edge 2 60000 1.77 0.08 0.81 0.08 ``` The figure at the top puts the twelve points on the in-sample versus out-of-sample plane. Three readings: - **Control.** At 60,000 steps, the three agents have an in-sample Sharpe of 0.45 to 0.73 and barely trade (turnover of 0.02 to 0.08): they have learned to be long, the chance drift of 2008-2017. Out of sample, −0.53 to +0.03. At 30,000 steps they were still trading (turnover of 0.2 per day) following patterns in the noise, and out of sample they lost between −0.65 and −1.17. No control agent has learned anything, and in sample they all seemed to win. - **Edge.** The agents pick up the planted edge: out of sample, 0.63 to 0.81, above the naive oracle (0.58) because they learn to trade less (turnover of 0.08 to 0.11 against 0.26). It is the only good thing in the table, and it is possible because there is something to learn and the costs are set correctly in the reward. - **The gap.** In sample, the agents on the series with an edge promise 1.77 to 1.98: more than twice what they deliver out of sample. Training more raises the in-sample Sharpe for all three seeds (from 30,000 to 60,000 steps); the out-of-sample one also rises in this configuration, but exercise 1 shows that this does not last. With real data, only the in-sample column would be seen if validation were done badly, and the difference between a Sharpe of 1.9 and 0.7 is the difference between an exceptional strategy and an ordinary one. ## The seed is also a hyperparameter Six agents, six results. With real data nobody knows what the oracle is, and the temptation is to present the agent with the best in-sample Sharpe or, worse, the one with the best out-of-sample Sharpe after having seen all six. Both are multiple tests (lesson 2.5.4). The standard error of an annual Sharpe measured over $T$ years is about $\sqrt{(1 + SR^2/2)/T}$, and the maximum of $N$ Sharpes with no edge at all has a positive expectation: ```python from scipy.stats import norm def error_sharpe(sr, anos): return np.sqrt((1 + sr**2 / 2) / anos) def maximo_esperado(n, error): """Expected maximum of n Sharpes with mean 0 (Bailey and López de Prado; lesson 2.5.4).""" g = 0.5772156649 # Euler-Mascheroni constant return error * ((1 - g) * norm.ppf(1 - 1 / n) + g * norm.ppf(1 - 1 / (n * np.e))) anos_dentro, anos_fuera = corte / 252, (len(s) - corte) / 252 print(f"standard error of the Sharpe: {error_sharpe(0, anos_dentro):.2f}" f" over {anos_dentro:.0f} years, {error_sharpe(0, anos_fuera):.2f}" f" over {anos_fuera:.0f} years") for n in (3, 6, 20): print(f"expected maximum of {n:>2} Sharpes with no edge (out): " f"{maximo_esperado(n, error_sharpe(0, anos_fuera)):.2f}") fin = res[res["steps"] == 60_000].set_index(["series", "seed"]) for k in SERIES: x = fin.loc[k] mejor = x["Sharpe in"].idxmax() print(f"{k:<8} out: mean {x['Sharpe out'].mean():+.2f}, " f"range {x['Sharpe out'].min():+.2f} to " f"{x['Sharpe out'].max():+.2f}; the best in sample " f"(seed {mejor}) gives {x.loc[mejor, 'Sharpe out']:+.2f}") ``` ```output standard error of the Sharpe: 0.32 over 10 years, 0.35 over 8 years expected maximum of 3 Sharpes with no edge (out): 0.30 expected maximum of 6 Sharpes with no edge (out): 0.46 expected maximum of 20 Sharpes with no edge (out): 0.67 control out: mean -0.23, range -0.53 to +0.03; the best in sample (seed 1) gives -0.53 edge out: mean +0.75, range +0.63 to +0.81; the best in sample (seed 1) gives +0.80 ``` With a standard error of 0.35 over 8 years, the dispersion across seeds on the series with an edge (0.63 to 0.81) is small compared with the error of the sample itself: the three seeds see the same 2,011 test days, and if those days had been different, the three figures would move together. On the control, the dispersion is larger (−0.53 to +0.03) and choosing by the in-sample Sharpe leads to the worst out-of-sample seed (seed 1). The table of expected maxima gives the scale: with 6 agents with no edge at all, the expected best out-of-sample Sharpe is already 0.46, and with 20 configurations (a few architectures, learning rates and seeds, something any project reaches in an afternoon), 0.67. An out-of-sample 0.7 chosen among twenty attempts proves nothing; here it proves something only because the control is known. The Deflated Sharpe of lesson 2.5.4 formalizes the correction; Henderson et al. (2018) document the same seed problem in RL papers in general, outside finance. ## Reward hacking: a bug in the costs The last experiment is a very common programming bug. Whoever writes the environment charges the cost “when opening a position”: if you are flat and enter, you pay; if you are already in, you do not. It seems reasonable until you think about the flip: going from long to short, which in reality costs twice as much as opening, is free, and so is closing. It is the `coste_solo_al_abrir=True` variant of the environment. An agent is trained with it on the series with an edge (seed 0) and evaluated, as always, with the correct costs. ```python m_hack, hist_hack = entrena(SERIES["edge"], 0, coste_solo_al_abrir=True) honesto = res[(res["series"] == "edge") & (res["seed"] == 0)] comp = pd.DataFrame( [["correct cost", *honesto.iloc[-1, 3:]], ["cost only on opening", *hist_hack[-1][1:]]], columns=["training reward", "Sharpe in", "turn. in", "Sharpe out", "turn. out"]) def sharpe_segun_recompensa(m, r): """In-sample Sharpe with the costs of the hacked environment: what the agent believes it earns.""" env = DireccionalEnv(r, ventana=corte - L - 1, coste_solo_al_abrir=True) o, _ = env.reset(seed=0) env.t, env.t_fin, env.pos = L, corte - 1, 0.0 o, pnl, fin = env._obs(), [], False while not fin: a = m.predict(o, deterministic=True)[0] o, rec, _, fin, _ = env.step(a) pnl.append(rec * SD) pnl = np.array(pnl) return round(pnl.mean() / pnl.std() * np.sqrt(252), 2) print("In-sample Sharpe according to the hacked reward:", sharpe_segun_recompensa(m_hack, SERIES["edge"])) comp ``` ```output In-sample Sharpe according to the hacked reward: 2.03 training reward Sharpe in turn. in Sharpe out turn. out 0 correct cost 1.84 0.11 0.63 0.11 1 cost only on opening 1.48 0.40 0.37 0.34 ``` *Fig. — Reward hacking: the agent trained with the cost charged only when opening believes it earns more than the honest one; with real costs it turns over almost four times as much and earns less in and out of sample.* Horizontal bars of annual Sharpe: according to the rigged reward 2.03 (highlighted); the same agent with real costs 1.48 in-sample (turnover 0.40 a day) and 0.37 out-of-sample (0.34); the agent with the correct cost 1.84 in-sample and 0.63 out-of-sample (turnover 0.11). The agent with the hacked reward believes it earns an in-sample Sharpe of 2.03, more than the honest one. With the real costs, 1.48 in sample and 0.37 out of sample, against 1.84 and 0.63 for the agent with the cost programmed correctly. The figure compares the two. The cause is the turnover: 0.40 units per day in sample, almost four times that of the honest agent (0.11). For it, flipping from long to short and closing cost nothing, so it follows every change in the signal and in the noise of the lags; the evaluation with the real costs charges it for all those flips. This is *reward hacking* (or *specification gaming*): the agent optimizes the reward it was given, not the one it was meant to be given, and it is better than a person at finding the loophole. Two defenses, both cheap. Always evaluate with a P&L calculation independent of the training environment (like `evalua` in this lesson), and watch the turnover: an agent that turns over much more than the oracle, or than a simple rule with the same information, is exploiting something. A unit test of the environment with known flips (from +1 to −1 must cost twice the unit cost) would have caught this bug before training; `check_env` would not, because it checks the interface, not the accounting. ## FinRL as a critical reference FinRL is the best-known RL library for trading: it is maintained by the AI4Finance Foundation, described in a 2020 paper (Liu et al., arXiv:2011.09607) and offers environments with the Gymnasium interface for stocks, crypto and portfolios, agents from Stable-Baselines3, ElegantRL and RLlib, data downloaders and end-to-end tutorials. What was checked in its sources as of October 23, 2026: | Item | Value | Source | |---|---|---| | License | MIT | GitHub (repository API) and PyPI | | Latest version on PyPI | 0.3.7, released on April 12, 2024 | `pypi.org/pypi/finrl/json` | | Latest tagged version on GitHub | v0.3.8, March 20, 2026 | GitHub releases API | | Activity | last commit on `master` on October 20, 2026 | GitHub commits API | | `StockTradingEnv` reward | change in the total account value at each step, times `reward_scaling` | `finrl/meta/env_stock_trading/env_stocktrading.py` | The PyPI package is two and a half years behind the repository: `pip install finrl` installs code from 2024. It is not installed in the course environment; the typical flow of its tutorials, with its current API as read in the repository, is this (it needs network access to download prices from Yahoo): ```python skip from finrl.agents.stablebaselines3.models import DRLAgent from finrl.config import INDICATORS from finrl.meta.env_stock_trading.env_stocktrading import ( StockTradingEnv) from finrl.meta.preprocessor.preprocessors import (FeatureEngineer, data_split) from finrl.meta.preprocessor.yahoodownloader import YahooDownloader df = YahooDownloader(start_date="2009-01-01", end_date="2024-01-01", ticker_list=["AAPL", "MSFT", "JPM"]).fetch_data() df = FeatureEngineer(use_technical_indicator=True, tech_indicator_list=INDICATORS, use_turbulence=True).preprocess_data(df) train = data_split(df, "2009-01-01", "2020-07-01") trade = data_split(df, "2020-07-01", "2024-01-01") n = train["tic"].nunique() kw = dict(stock_dim=n, hmax=100, initial_amount=1_000_000, num_stock_shares=[0] * n, buy_cost_pct=[0.001] * n, sell_cost_pct=[0.001] * n, reward_scaling=1e-4, state_space=1 + 2 * n + len(INDICATORS) * n, action_space=n, tech_indicator_list=INDICATORS) env_train, _ = StockTradingEnv(df=train, **kw).get_sb_env() agente = DRLAgent(env=env_train) ppo = agente.get_model("ppo") ppo = DRLAgent.train_model(ppo, tb_log_name="ppo", total_timesteps=50_000) cuenta, acciones = DRLAgent.DRL_prediction( model=ppo, environment=StockTradingEnv(df=trade, **kw)) ``` None of this is wrong in itself: it is a convenient way to set up the experiment. The problem is what it does not do, and a tutorial result being read as evidence: - **A single** training-test **split**, with no walk-forward (lesson 5.4.3) or purging; if the hyperparameters or the number of steps are tuned by looking at the test period, the test stops being one. - **One seed** by default (`seed=None` in `get_model`): the result changes on every run, and without several seeds you do not know by how much (this module: several tenths of Sharpe). - **A universe of current components**: in October 2026 the repository itself merged *pull request* 1462, which clarifies the survivorship bias of a tutorial's Dow backtest and switches to point-in-time components. Results published earlier with that tutorial carry the bias inside them (lessons 1.4.4 and 2.5.1). - **No control**: no tutorial compares with the same agent trained on a series with no signal, which is what separates learning an edge from learning the sample. - **A comparison with buy and hold or with an index** over a specific period, with no standard error: with 3 years of test data, the standard error of the Sharpe is almost 0.6. ## Before believing a directional agent 1. **Is there a control?** The same agent, with the same configuration, trained on a series with no signal (permuted or synthetic) has to give out of sample what is expected from chance. 2. **Truly out of sample?** A test after the training and after the hyperparameter choice, ideally in walk-forward with retraining (lesson 5.4.3). 3. **How many seeds and configurations?** All of them, with their dispersion, and the Sharpe deflated by the number of tests (lesson 2.5.4). 4. **Full net P&L?** Computed outside the training environment, with real costs, slippage and financing, and the turnover in plain view. 5. **Against what?** Buy and hold, the simplest linear rule with the same information (5.5.1) and, if one exists, an analytical benchmark. An agent that does not beat a regression with the same features has learned nothing worth its complexity. 6. **Point-in-time data?** A universe with the companies that existed on each date and features with the date on which they were known (lesson 5.2.2). ## Exercises 1. Repeat the training on the series with an edge, seed 0, with twice as many steps (four blocks of 30,000). Does the in-sample Sharpe go up? And the out-of-sample one? **Solution.** ```python _, hist_largo = entrena(SERIES["edge"], 0, bloques=4) pd.DataFrame(hist_largo, columns=["steps", "Sharpe in", "turn. in", "Sharpe out", "turn. out"]) ``` ```output steps Sharpe in turn. in Sharpe out turn. out 0 30000 1.60 0.14 0.38 0.15 1 60000 1.84 0.11 0.63 0.11 2 90000 2.10 0.11 0.71 0.12 3 120000 2.06 0.15 0.55 0.16 ``` In sample, the Sharpe rises to 2.10 at 90,000 steps and stays at 2.06 at 120,000. Out of sample, it rises to 0.71 and falls to 0.55, with more turnover (0.16 per day): the agent starts fitting details of the training sample that do not repeat. If the number of steps were chosen by looking at the test period (90,000, the best), that choice would be another multiple test, and the 0.71 would stop being an out-of-sample measurement. In an honest validation, the number of steps is chosen with a validation inside the training period (walk-forward or purged k-fold, lessons 5.4.2 and 5.4.3) and the test period is looked at only once. 2. Using the standard error of the Sharpe, how many years of data would be needed for the out-of-sample Sharpe of the oracle on the series with an edge (the expected one, not the realized one) to be two standard errors above zero? **Solution.** ```python # Expected annual Sharpe of the gross oracle: kappa · E|s| · sqrt(252) sr = KAPPA * np.sqrt(2 / np.pi) * np.sqrt(252) neto = sr - COSTE * 0.2 / SD * np.sqrt(252) # turnover ≈ 0.2/day for nombre, x in (("gross", sr), ("net-of-cost", neto)): anos = 4 * (1 + x**2 / 2) / x**2 print(f"Expected {nombre} Sharpe: {x:.2f}; years for 2 standard " f"errors: {anos:.0f}") ``` ```output Expected gross Sharpe: 1.27; years for 2 standard errors: 4 Expected net-of-cost Sharpe: 0.98; years for 2 standard errors: 6 ``` With a true IC of 0.10, the oracle has an expected gross Sharpe of 1.27 and, with a turnover of about 0.2 units per day and a 5 bp cost, 0.98 net. For the realized Sharpe to be two standard errors from zero, 4 years are needed gross and 6 net. With an edge half that size (an IC of 0.05, more realistic) the gross Sharpe drops to 0.63 and the costs do not go down: about 12 years would be needed gross and about 35 net. That is why the 8 test years of this lesson are enough for the planted edge (and even so the realized oracle, 0.58, ends up more than one standard error below its expectation), and why, with real signals with an IC of a few hundredths, a backtest of a few years cannot tell a good agent from a lucky one. 3. A paper presents an RL agent that, over five years of testing on 30 stocks, obtains a Sharpe of 2.1 against 0.8 for buy and hold. It does not say how many configurations or how many seeds it tried. Which three pieces of information would you ask for, and why? **Solution.** First, the number of configurations and seeds tried and the dispersion of their results: with N tests, the expected maximum with no edge grows with N (with a standard error of 0.45 over five years and N = 20, it is already around 0.86 of Sharpe over the benchmark), and deflation is needed (lesson 2.5.4). Second, how the universe of 30 stocks and the hyperparameters were chosen: if the stocks are the ones that make up an index today, there is survivorship bias (the one FinRL fixed in its Dow tutorial); if the number of steps or the architecture was chosen by looking at the test period, there is no test period. Third, the full net P&L with the turnover: an agent that turns over the portfolio daily with 0.1% costs loses about 25 points of return per year for each unit of daily turnover (0.1% × 252 sessions), and the reward hacking in this lesson shows that a reward with badly programmed costs produces exactly that turnover. With those three pieces of information, and a control with no signal, the 2.1 can start to be discussed. ## Self-assessment 1. On the control series (returns with a constant conditional mean equal to zero), the three PPO agents end with an in-sample Sharpe between 0.45 and 0.73. What explains that figure? - A. The agent has found an edge the README does not know about - B. PPO always gives a positive Sharpe - C. The training sample had, by chance, a positive drift (buy and hold gives 0.60 in those years) and the agent learned to be long; out of sample the drift does not exist - D. Costs are not charged in sample *Answer C: The training sample had, by chance, a positive drift (buy and hold gives 0.60 in those years) and the agent learned to be long; out of sample the drift does not exist.* With 10 years of data, the standard error of an annual Sharpe is about 0.3: a Sharpe of 0.6 on an asset with no drift is a sample accident of two standard errors, and the agent learns it as if it were a rule. 2. On the series with an edge (true IC 0.10), the agents give an in-sample Sharpe of 1.77 to 1.98 and an out-of-sample one of 0.63 to 0.81. Which reading is correct? - A. The agent stopped working in 2018 - B. The in-sample Sharpe mainly measures the fit to the sample (including the luck of a measured IC of 0.122); the out-of-sample one measures what was really learned, which is real here because the edge is planted - C. You have to train more until the two match - D. The out-of-sample result is chance, because it beats the oracle *Answer B: The in-sample Sharpe mainly measures the fit to the sample (including the luck of a measured IC of 0.122); the out-of-sample one measures what was really learned, which is real here because the edge is planted.* The agents beat the naive oracle (0.58) because they trade less, not by chance. But the in-sample Sharpe promises more than twice what they deliver: with real data, that gap is what separates an exceptional strategy from an ordinary one. 3. Six agents are trained (3 seeds × 2 configurations) and the one with the best in-sample Sharpe is presented. Which correction applies? - A. None: the seed is not a parameter - B. Divide the Sharpe by 6 - C. Repeat the training with the same seed - D. Treat the seeds as multiple tests: compare the chosen Sharpe with the expected maximum of 6 Sharpes with no edge (Deflated Sharpe, lesson 2.5.4) *Answer D: Treat the seeds as multiple tests: compare the chosen Sharpe with the expected maximum of 6 Sharpes with no edge (Deflated Sharpe, lesson 2.5.4).* Choosing the best of N seeds is choosing the maximum of N random variables. With a standard error of 0.35 (8 years) and N = 6, the expected maximum with no edge at all is already 0.46 of annual Sharpe. 4. The training reward charges the cost only when opening a position from flat; flipping from long to short is free. What does the agent do? - A. It exploits the bug: it turns over almost four times more than the honest agent, because for it flipping and closing cost nothing; it believes it earns a Sharpe of 2.03 and, with the real costs, gives 0.37 out of sample against 0.63 - B. Nothing different, because the costs are small - C. It stops trading - D. It always stays out of the market *Answer A: It exploits the bug: it turns over almost four times more than the honest agent, because for it flipping and closing cost nothing; it believes it earns a Sharpe of 2.03 and, with the real costs, gives 0.37 out of sample against 0.63.* This is reward hacking: the agent optimizes the reward it is given, not the one it was meant to be given. The only defense is to evaluate with the full net P&L, computed outside the training environment. 5. Which check matters most before believing a backtest of a directional RL agent built with FinRL or any other library? - A. That it uses PPO and not DQN - B. That the test period lies outside the training period and the hyperparameter choice, with a point-in-time universe, realistic costs, several seeds and a comparison with buy and hold and with a control with no signal - C. That the equity curve is smooth - D. That it has many stars on GitHub *Answer B: That the test period lies outside the training period and the hyperparameter choice, with a point-in-time universe, realistic costs, several seeds and a comparison with buy and hold and with a control with no signal.* The library validates nothing for the user. FinRL itself fixed, in October 2026, the survivorship bias of a tutorial that used the current Dow components. ## Key points - Directional trading is the worst case for RL: a single history that cannot be simulated again, a signal of a few hundredths of IC, a market that changes and a network with thousands of parameters. - On the control with no predictability, the PPO agents learn the chance drift of the training sample (in-sample Sharpe of 0.45 to 0.73, barely trading) and out of sample give −0.53 to +0.03. - With a planted edge of IC 0.10, the agents pick it up and learn to trade less than the naive oracle: out of sample, 0.63 to 0.81 against 0.58. In sample they promised 1.77 to 1.98, more than twice as much. - The seed is a hyperparameter: choosing the best of N is a multiple test, and the Sharpe has to be deflated (lesson 2.5.4). - Reward hacking: with the cost programmed wrong (free when flipping and closing), the agent turns over almost four times more and its out-of-sample Sharpe falls from 0.63 to 0.37, while its reward told it 2.03. Always evaluate with the full P&L, outside the training environment. - FinRL (MIT license) is useful as a catalog of environments and for learning, not as evidence. On PyPI its latest version is 0.3.7, from April 2024, while the repository is still active; its tutorials are no substitute for honest validation. ## Resources ### Research - [Intraday Seasonality IV - Field Search](https://pythonparatrading.com/en/seasonality-intraday-optimization/) · Research-0027 · 7 min ### Documentation - [Henderson, P. et al. (2018): Deep Reinforcement Learning that Matters. AAAI. arXiv:1709.06560](https://arxiv.org/abs/1709.06560) · arxiv.org - [Moody, J. and Saffell, M. (2001): Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks 12(4)](https://doi.org/10.1109/72.935097) · doi.org - [Liu, X.-Y. et al. (2020): FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance. arXiv:2011.09607](https://arxiv.org/abs/2011.09607) · arxiv.org - [FinRL on GitHub (AI4Finance-Foundation, MIT license)](https://github.com/AI4Finance-Foundation/FinRL) · github.com - [FinRL, pull request 1462: survivorship bias in the Dow backtest](https://github.com/AI4Finance-Foundation/FinRL/pull/1462) · github.com ### Data - [Level 5 data: returns and realized variance of six assets (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/vol5_diario.parquet) · Download · vol5_diario.parquet Previous lesson: [5.9.2 · RL for execution, market making and dynamic hedging](https://pythonparatrading.com/en/python-trading-course/level-5/rl-execution-market-making-hedging/) Next lesson: [5.10.1 · SHAP, partial dependence and the stability of feature importances](https://pythonparatrading.com/en/python-trading-course/level-5/shap-partial-dependence-stability/)