Performance: measure, profile and compile only what is needed.

Measure before optimizing

The slow code in a trading system is almost never the strategy's arithmetic. It is usually the way the data are traversed: one pandas row at a time, a conversion on every iteration, a disk read inside a loop. Guessing where the problem is, is the most common way to lose an afternoon optimizing what does not matter. The method is always the same:

flowchart TB
  M[Measure] --> P[Profile]
  P --> A[Algorithm]
  A --> C[Compile]
  C --> M2[Measure again]

First measure the whole program with realistic data; then profile it to find out which part takes the time; change the algorithm or the data structure (almost always what pays off most) and, only if that is still not enough, compile the hot part. Then measure again, because the bottleneck moves. How much can be gained is bounded by Amdahl's law: if a part takes the fraction $p$ of the time and is sped up $s$ times, the whole program is sped up

With $p = 0.9$, speeding that part up 100 times gives $S \approx 9.2$; not even with an infinite $s$ does it exceed 10. The figure at the end of the lesson plots the curve for several values of $p$.

The lesson's case is an intraday breakout strategy on the 96,330 one-minute bars of the level 5 synthetic future (hf5_barras_1min.parquet): it buys when the close exceeds the high of the previous 30 bars; it exits when the close falls more than 4 ATR below the highest close reached since the entry (a trailing stop) or when a new session starts. The ATR here is a 14-bar exponential average of each bar's range (high minus low). The exit depends on the path (the highest close since the entry, whether the position is open or not), so it cannot be written with column operations: it needs a loop, as in lesson 2.4.2. This is how someone coming from a notebook would write it:

import time

import numpy as np
import pandas as pd

df = pd.read_parquet("hf5_barras_1min.parquet")
N, K, ALFA = 30, 4.0, 2 / 15      # 30-bar breakout, 4-ATR stop

def backtest_ingenuo(df, n=N, k=K, alfa=ALFA):
    """Position (0 or 1) on each bar."""
    pos, pico, atr = 0, 0.0, None
    salida = []
    for i in range(len(df)):
        fila = df.iloc[i]
        rango = fila["maximo"] - fila["minimo"]
        atr = rango if atr is None else alfa * rango + (1 - alfa) * atr
        if i < n or fila["ts"].date() != df["ts"].iloc[i - 1].date():
            pos = 0                               # new session
        elif pos == 0:
            techo = df["maximo"].iloc[i - n:i].max()
            if fila["cierre"] > techo:
                pos, pico = 1, fila["cierre"]     # entry
        else:
            pico = max(pico, fila["cierre"])
            if fila["cierre"] < pico - k * atr:
                pos = 0                           # trailing stop
        salida.append(pos)
    return np.array(salida, dtype=np.int8)

sub = df.iloc[:20_000]
t0 = time.perf_counter()
pos_ref = backtest_ingenuo(sub)
t_ingenuo = time.perf_counter() - t0          # not printed
entradas = int((np.diff(pos_ref, prepend=0) == 1).sum())
print(len(df), "bars,", df["ts"].dt.date.nunique(), "sessions")
print(f"first 20,000: {entradas} entries, "
      f"in the market {pos_ref.mean():.1%} of the bars")

96330 bars, 247 sessions
first 20,000: 226 entries, in the market 40.7% of the bars

On the machine the lesson was written on, the first 20,000 bars take about 3.7 seconds: about 185 microseconds per bar, about 18 seconds for the whole file. The cells do not print timings because they change from one machine to another and from one run to another; they store the time in a variable and, when comparing, print only whether one version is faster than another by a wide margin. For a backtest that runs once, 18 seconds do not matter. For a sweep of 200 parameter combinations (lesson 2.5.6), or for a hundred assets, they are an hour.

cProfile: exact counts

cProfile is the standard library's deterministic profiler: it intercepts every function call and return and records how many times each one was called and how long it took. Its default report (pstats) is long; what matters here are the functions that backtest_ingenuo calls directly, with their number of calls, sorted by cumulative time. 5,000 bars are enough:

import cProfile
import pstats

perfil = cProfile.Profile()
perfil.runcall(backtest_ingenuo, df.iloc[:5_000])
st = pstats.Stats(perfil)
raiz = next(f for f in st.stats if f[2] == "backtest_ingenuo")
total = st.stats[raiz][3]                     # cumulative time
hijos = []
for (fichero, _, nombre), (*_, llamadores) in st.stats.items():
    if raiz in llamadores:
        _, ncalls, _, acumulado = llamadores[raiz]
        corto = fichero.rsplit("site-packages/", 1)[-1]
        hijos.append((acumulado, ncalls, f"{corto}:{nombre}"))
hijos.sort(reverse=True)
for acumulado, ncalls, nombre in hijos[:2]:
    print(f"{ncalls:>7}  {nombre}")
pandas = sum(a for a, _, n in hijos if n.startswith("pandas"))
print("pandas, more than 80% of the time:", pandas / total > 0.8)

  13038  pandas/core/indexing.py:__getitem__
   8038  pandas/core/frame.py:__getitem__
pandas, more than 80% of the time: True

The two functions that accumulate the most time are pandas' indexing functions: iloc[...] and df[...]. The counts can be reconstructed by reading the code, and it is worth doing: the 13,038 calls to _iLocIndexer.__getitem__ are the 5,000 from df.iloc[i], the 4,970 from df["ts"].iloc[i - 1] (every bar except the first 30) and 3,068 from df["maximo"].iloc[i - n:i] (only when out of the market); the 8,038 calls to DataFrame.__getitem__ are the df["ts"] and df["maximo"] of the last two. On the author's machine, iloc[...] takes more than half of the time, df[...] more than a quarter, and the strategy's arithmetic (the function's [self] in the pyinstrument report below) around 4%.

cProfile has two drawbacks worth knowing. It adds a fixed cost to every Python call, so it distorts code with many small calls (like this one) more; the percentages are indicative and the absolute times higher than without profiling. And it only measures the thread it is enabled from. What it gives in exchange are exact counts: if a function is called a million times when it should be called a thousand, cProfile says so without ambiguity.

Sampling profilers: pyinstrument and py-spy

A sampling profiler does not intercept calls: it looks at the program's stack every so often (every millisecond, for example) and counts which functions it was in. A function that appears in 50% of the samples took, approximately, half of the time. It loses the exact counts and very short functions, but it interferes much less with the program.

pyinstrument samples from inside the process and presents a call tree. Its text output carries the time of day, the number of samples and the seconds of each branch, all variable; the cell walks the tree and writes only the branches under backtest_ingenuo that exceed 20% of the time:

from pyinstrument import Profiler
from pyinstrument.renderers import ConsoleRenderer

pf = Profiler(interval=0.001)
pf.start()
backtest_ingenuo(df.iloc[:5_000])
pf.stop()
arbol = ConsoleRenderer().preprocess(pf.last_session.root_frame())

def buscar(marco, nombre):
    if marco.function == nombre:
        return marco
    for hijo in marco.children:
        if (r := buscar(hijo, nombre)) is not None:
            return r

b = buscar(arbol, "backtest_ingenuo")
for hijo in b.children:
    if hijo.time / b.time > 0.2:
        nombre = f"{hijo.class_name}.{hijo.function}"
        print(nombre, hijo.file_path_short)

_iLocIndexer.__getitem__ pandas/core/indexing.py
DataFrame.__getitem__ pandas/core/frame.py

The same two branches as cProfile. pyinstrument's full report, as printed by output_text() when running this same code as a script (pyi.py) on the author's machine (the profile's path shortened; the timings change on every run), shows the grouped tree, with pandas' internal frames collapsed:

print(pf.output_text(unicode=True, color=False))

  _     ._   __/__   _ _  _  _ _/_   Recorded: 23:43:42  Samples:  1774
 /_//_/// /_\ / //_// / //_'/ //     Duration: 1.774     CPU time: 1.753
/   _/                      v5.1.3

Profile at pyi.py:33

1.773 <module>  pyi.py:1
└─ 1.773 backtest_ingenuo  pyi.py:9
   ├─ 0.962 _iLocIndexer.__getitem__  pandas/core/indexing.py:1192
   │     [38 frames hidden]  pandas, typing, annotationlib, <built...
   ├─ 0.498 DataFrame.__getitem__  pandas/core/frame.py:4337
   │     [20 frames hidden]  pandas, typing, annotationlib
   ├─ 0.128 wrapper  pandas/util/_decorators.py:328
   │     [15 frames hidden]  pandas, numpy, <built-in>
   ├─ 0.112 Series.__getitem__  pandas/core/series.py:945
   │     [5 frames hidden]  pandas
   └─ 0.067 [self]  pyi.py

py-spy samples from outside the process: it is a separate program (written in Rust) that reads the interpreter's memory. That is why the code does not have to be touched and it works on a bot that is already running. On Linux, reading another process's memory requires permissions: with kernel.yama.ptrace_scope = 1, the usual setting, only a child process can be sampled, so py-spy launches the script itself. The cell writes a script with the same backtest (extracted with inspect.getsource), runs it under py-spy at 200 samples per second and counts, in the raw format (one line per distinct stack: the frames separated by ; and the number of samples), which of the script's functions is deepest in the stack in each sample, not counting the module level (<module>):

import inspect
import subprocess
from collections import Counter
from pathlib import Path

Path("perfilar.py").write_text(
    "import numpy as np\nimport pandas as pd\n\n"
    "N, K, ALFA = 30, 4.0, 2 / 15\n\n"
    + inspect.getsource(backtest_ingenuo)
    + "\n\ndef cargar():\n"
    "    return pd.read_parquet('hf5_barras_1min.parquet')\n\n"
    "backtest_ingenuo(cargar().iloc[:5_000])\n")
r = subprocess.run(["py-spy", "record", "--nonblocking", "-r", "200",
                    "-f", "raw", "-o", "perfil.txt", "--",
                    "python", "perfilar.py"],
                   capture_output=True, text=True)
cuenta = Counter()
for linea in Path("perfil.txt").read_text().splitlines():
    pila, n = linea.rsplit(" ", 1)
    propios = [m.split(" (")[0] for m in pila.split(";")
               if "(perfilar.py:" in m and "<module>" not in m]
    if propios:
        cuenta[propios[-1]] += int(n)
print("py-spy:", r.returncode, "| most sampled own function:",
      cuenta.most_common(1)[0][0])

py-spy: 0 | most sampled own function: backtest_ingenuo

With Python 3.14 and py-spy 0.4.2, on this machine, many --nonblocking samples come out empty or truncated: the process keeps running while py-spy reads its stack, and sometimes it reads it halfway. That is why the cell discards the module level, where the truncated stacks end up (along with the pandas import), and compares only the script's functions. The blocking mode, which stops the process at each sample, is reliable but slows this script down dozens of times. For coarse questions (which of my functions takes the time) it is enough; for fine percentages, pyinstrument or cProfile are better.

Python 3.15's sampling profiler

Python 3.15 (released on October 9, 202611Note 1 · Python.org, “Python 3.15.0” (https://www.python.org/downloads/release/python-3150/): release date, October 9, 2026. Accessed on October 23, 2026.) reorganizes the profilers into a new package, profiling (PEP 79922Note 2 · PEP 799, “A dedicated profiling package for organizing Python profiling tools” (https://peps.python.org/pep-0799/). Accessed on October 23, 2026.): profiling.tracing is the usual cProfile under another name, the old profile module (in pure Python) is deprecated, and profiling.sampling, nicknamed Tachyon, is a sampling profiler like py-spy but included in the interpreter33Note 3 · Python 3.15 documentation, “profiling.sampling — Statistical profiler” (https://docs.python.org/3.15/library/profiling.sampling.html): modes, default sampling rate, the run, attach, dump and replay commands, output formats and platform requirements. Accessed on October 23, 2026.. According to its documentation:

• It reads the stack from outside the process, without stopping it, at a cost to the program that the documentation describes as virtually zero. By default it takes 1,000 samples per second (-r); the duration can be limited (-d).
• It has four modes: wall-clock time (--mode=wall, the default, includes network and disk waits), CPU (--mode=cpu, only when the thread is executing), GIL (--mode=gil, only when it holds the GIL: separates Python from C extensions) and exceptions.
• It can launch a script (run), attach to a process by its PID (attach) or read a single snapshot of the stack (dump, useful with a hung process), and it writes pstats, collapsed stacks (the same format as py-spy's raw), HTML flame graphs, per-line heat maps or the Firefox Profiler format.
• On Linux, attach and dump have the same ptrace_scope restriction as py-spy; the profiler and the process must be the same minor Python version (and the same exact version if either is a pre-release).

The course environment is Python 3.14 (uv, on this machine, only offers 3.15.0rc2), so the following code is not run. With Python 3.15, the py-spy cell can be rewritten without installing anything, and the count is the same because the collapsed format is the same:

# Python 3.15: standard library sampling
r = subprocess.run([sys.executable, "-m", "profiling.sampling",
                    "run", "--collapsed", "-r", "10khz",
                    "perfilar.py"], capture_output=True, text=True)
# one line per stack: "frame;frame;frame N", like py-spy's raw

# A running bot (same user, ptrace permissions):
# python -m profiling.sampling attach --mode=cpu -d 30 <PID>
# python -m profiling.sampling dump <PID>      # where is it stuck?

For a live system (lesson 4.5.1), dump answers a frequent question at three in the morning: “what is the process that is not responding doing right now?”. Until now that required py-spy or gdb.

First, take pandas out of the loop

The profile says the problem is asking pandas for a row and a slice of a column on every iteration. Whatever does not depend on the path is computed outside, with column operations: the high of the previous 30 bars is a rolling maximum shifted by one bar, and the session change is comparing the date with the previous bar's. The loop receives only NumPy arrays:

flowchart LR
  P[prepare: pandas] --> A[NumPy arrays]
  A --> B[compilable loop]
  B --> S[positions]

def preparar(df, n=N):
    techo = df["maximo"].rolling(n).max().shift().to_numpy()
    dia = df["ts"].dt.floor("D")
    nueva = (dia != dia.shift()).to_numpy()
    return (df["maximo"].to_numpy(), df["minimo"].to_numpy(),
            df["cierre"].to_numpy(), techo, nueva)

def bucle(maximo, minimo, cierre, techo, nueva, n=N, k=K, alfa=ALFA):
    m = len(cierre)
    salida = np.zeros(m, dtype=np.int8)
    pos, pico, atr = 0, 0.0, 0.0
    for i in range(m):
        rango = maximo[i] - minimo[i]
        atr = rango if i == 0 else alfa * rango + (1 - alfa) * atr
        if i < n or nueva[i]:
            pos = 0
        elif pos == 0:
            if cierre[i] > techo[i]:
                pos, pico = 1, cierre[i]
        else:
            if cierre[i] > pico:
                pico = cierre[i]
            if cierre[i] < pico - k * atr:
                pos = 0
        salida[i] = pos
    return salida

import timeit

def medir(f, *args, veces=1, rep=5):
    """Best time per call over `rep` repetitions."""
    return min(timeit.repeat(lambda: f(*args), number=veces,
                             repeat=rep)) / veces

arrays = preparar(sub)
print("identical to the naive version:",
      np.array_equal(bucle(*arrays), pos_ref))
t_arrays = medir(bucle, *arrays)
print("more than 50 times faster:", t_ingenuo / t_arrays > 50)

identical to the naive version: True
more than 50 times faster: True

Same result, bar by bar, and more than 50 times faster (on the author's machine, about 280 times). The comparison with the reference result is the important part: an optimization that changes a single position is not an optimization, it is a different backtest. Two equivalence details: the techo shifted by one bar reproduces iloc[i - n:i], which does not include the current bar, and over the first 30 bars it is NaN, which is never less than a close, just as in the original version, which does not evaluate them.

The loop is still Python: each maximo[i] creates a Python float object from the array, each addition creates another. That is why, curiously, the same loop with Python lists (arr.tolist()) runs somewhat faster still than with arrays: NumPy is fast with operations on whole arrays, not with individual elements. To go further you have to leave the interpreter.

Numba: the same loop, compiled

Numba compiles Python functions to machine code with LLVM at the time of the first call, based on the argument types44Note 4 · Numba, “Compiling Python code with @jit” (https://numba.readthedocs.io/en/stable/user/jit.html). Accessed on October 23, 2026.. In nopython mode (the one used by @njit) the result does not use the interpreter at all: numbers are machine numbers and arrays, contiguous memory. The bucle function does not have to be rewritten:

from numba import njit

bucle_nb = njit(cache=True)(bucle)

completo = preparar(df)
pos_nb = bucle_nb(*completo)                  # compiles here
pos_py = bucle(*completo)
print("identical to the Python loop on the 96,330 bars:",
      np.array_equal(pos_nb, pos_py))
t_py = medir(bucle, *completo, rep=3)
t_nb = medir(bucle_nb, *completo, veces=20)
print("Numba, more than 20 times faster than the Python loop:",
      t_py / t_nb > 20)

identical to the Python loop on the 96,330 bars: True
Numba, more than 20 times faster than the Python loop: True

On the author's machine, about 240 times: the loop over the 96,330 bars goes from about 58 milliseconds to about 0.24. The first call compiles (around half a second for a function like this) and cache=True stores the result on disk, so that the next process does not compile again as long as neither the code nor the types change. That is why the cell measures after the first call.

What Numba cannot compile fails with a typing error, not with a slow program. Passing it the DataFrame instead of the arrays:

from numba.core.errors import TypingError

ingenuo_nb = njit(backtest_ingenuo)
try:
    ingenuo_nb(df.iloc[:100])
except TypingError as e:
    print(type(e).__name__ + ":", str(e).splitlines()[0])

TypingError: Failed in nopython mode pipeline (step: nopython frontend)

Numba understands numbers, NumPy arrays, tuples, some typed dictionaries and lists and a large part of NumPy; it does not understand pandas or arbitrary objects. That shapes the design: the boundary between pandas and the compiled code is made of arrays, as in preparar. Three more precautions. fastmath=True speeds up some operations by allowing floating-point sums to be reordered, which changes the last decimals: in a backtest that is compared against a reference result, better not to use it. parallel=True with prange distributes independent iterations across threads, and this loop has none (each bar depends on the previous one), but a parameter sweep does (exercise 1). And compiled code does not check out-of-range indices by default: an error that in Python would raise IndexError can read someone else's memory without warning.

Cython: declared types and C

Cython translates a dialect of Python with C types into a C extension compiled with the system compiler (here, gcc). Without type declarations, the translated code still calls the Python API on every operation and gains little; with types (cdef double, double[:] memoryviews over the arrays), the loop becomes pure C55Note 5 · Cython, “Typed Memoryviews” (https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html). Accessed on October 23, 2026.. The cell writes the typed version to a .pyx file, annotates it with cython -a alongside an untyped copy of bucle and counts, in each annotation, how many lines of the loop body call the Python API (the ones Cython highlights in yellow in the HTML):

import re
import sys

PYX = '''# cython: boundscheck=False, wraparound=False
import numpy as np


def bucle_cy(const double[:] maximo, const double[:] minimo,
             const double[:] cierre, const double[:] techo,
             const unsigned char[:] nueva, int n=30,
             double k=4.0, double alfa=2.0 / 15):
    cdef Py_ssize_t i, m = cierre.shape[0]
    cdef double rango, pico = 0.0, atr = 0.0
    cdef signed char pos = 0
    salida = np.zeros(m, dtype=np.int8)
    cdef signed char[:] s = salida
    for i in range(m):
        rango = maximo[i] - minimo[i]
        atr = rango if i == 0 else alfa * rango + (1 - alfa) * atr
        if i < n or nueva[i]:
            pos = 0
        elif pos == 0:
            if cierre[i] > techo[i]:
                pos = 1
                pico = cierre[i]
        else:
            if cierre[i] > pico:
                pico = cierre[i]
            if cierre[i] < pico - k * atr:
                pos = 0
        s[i] = pos
    return salida
'''
Path("bucle_cy.pyx").write_text(PYX)
Path("bucle_sin_tipos.pyx").write_text(
    "import numpy as np\n\n" + inspect.getsource(bucle)
    .replace("n=N, k=K, alfa=ALFA", "n=30, k=4.0, alfa=2 / 15"))

def lineas_con_python(nombre):
    """Loop-body lines with calls to the Python API."""
    subprocess.run(["cython", "-3", "-a", f"{nombre}.pyx"], check=True)
    html = Path(f"{nombre}.html").read_text()
    puntos = [int(p) for p in
              re.findall(r'class="cython line score-(\d+)"', html)]
    codigo = Path(f"{nombre}.pyx").read_text().splitlines()
    ini = next(i for i, l in enumerate(codigo) if "for i in" in l)
    fin = next(i for i, l in enumerate(codigo) if "return" in l)
    return sum(p > 0 for p in puntos[ini + 1:fin]), fin - ini - 1

for nombre in ("bucle_sin_tipos", "bucle_cy"):
    con, total = lineas_con_python(nombre)
    print(f"{nombre}: {con} of {total} loop lines use Python")

bucle_sin_tipos: 9 of 13 loop lines use Python
bucle_cy: 0 of 14 loop lines use Python

Without types, nine of the loop's thirteen lines go through Python objects; with types, none. The annotation is the working tool with Cython: you write, annotate, look at the yellow lines inside the loops and add types until they disappear. The directives on the first line remove the index checks (boundscheck) and the negative-index handling (wraparound): they are faster and as dangerous as in Numba. pyximport compiles the .pyx when it is imported, in a temporary folder:

import pyximport

sys.path.insert(0, ".")
pyximport.install(build_dir="build_pyx", language_level=3)
from bucle_cy import bucle_cy

maximo, minimo, cierre, techo, nueva = completo
pos_cy = bucle_cy(maximo, minimo, cierre, techo, nueva.view(np.uint8))
print("identical to Numba:", np.array_equal(pos_cy, pos_nb))
t_cy = medir(bucle_cy, maximo, minimo, cierre, techo,
             nueva.view(np.uint8), veces=20)
print("Cython, same order as Numba (between 0.2 and 5 times):",
      0.2 < t_nb / t_cy < 5)

identical to Numba: True
Cython, same order as Numba (between 0.2 and 5 times): True

Same result and the same order of magnitude as Numba. On the author's machine, with the whole file, Cython takes about 0.19 milliseconds and Numba about 0.24; with 20,000 bars, Cython comes out about 2.7 times faster. The differences between two compiled versions depend on the size and on details such as the fixed cost of each call; what matters is that both are hundreds of times ahead of the Python loop. nueva.view(np.uint8) reinterprets the boolean array as bytes without copying it, because Cython memoryviews do not accept bool directly. To distribute a package with Cython you do not use pyximport but the package's build system: a setup.py with cythonize or a backend such as setuptools declared in pyproject.toml, which compiles at install time and produces wheels per platform.

Numba or Cython? For a numerical research loop, Numba: it is Python, there is no separate compilation step and no compiler is needed on the machine. Cython is the better fit when you need to call C libraries, mix compiled code with Python objects or distribute an extension without the user installing LLVM; it is what pandas, scikit-learn or NautilusTrader use in parts of their core (lesson 2.4.5).

Rust with PyO3: what you need to know

Rust is a compiled language with memory safety guaranteed by the compiler: no Cython-style boundscheck=False and no silent out-of-range accesses (in Rust, an out-of-range index stops the program with an error). PyO3 generates the bindings to Python and maturin builds and publishes the wheels66Note 6 · PyO3 (https://pyo3.rs/) and its README, with the Cargo.toml example and #[pyo3::pymodule] for version 0.30; rust-numpy (https://github.com/PyO3/rust-numpy) for PyReadonlyArray1 and IntoPyArray; versions on crates.io. Accessed on October 23, 2026.. Half of the quantitative Python ecosystem's stack is already written this way: Polars (lesson 0.5.4), the NautilusTrader core, Ruff, uv and py-spy itself.

The same loop as a Rust extension. Compiling it requires downloading crates (Rust dependencies) from crates.io, and the course does not use the network, so the code is for reading only, with the versions published as of October 23, 2026 (pyo3 0.30.1, numpy 0.30.0):

# Cargo.toml
[package]
name = "bucle_rust"
version = "0.1.0"
edition = "2024"

[lib]
name = "bucle_rust"
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.30"
numpy = "0.30"

// src/lib.rs
#[pyo3::pymodule]
mod bucle_rust {
    use numpy::{IntoPyArray, PyArray1, PyReadonlyArray1};
    use pyo3::prelude::*;

    #[pyfunction]
    #[pyo3(signature = (maximo, minimo, cierre, techo, nueva,
                        n=30, k=4.0, alfa=2.0 / 15.0))]
    fn bucle_rs<'py>(
        py: Python<'py>,
        maximo: PyReadonlyArray1<'py, f64>,
        minimo: PyReadonlyArray1<'py, f64>,
        cierre: PyReadonlyArray1<'py, f64>,
        techo: PyReadonlyArray1<'py, f64>,
        nueva: PyReadonlyArray1<'py, bool>,
        n: usize, k: f64, alfa: f64,
    ) -> PyResult<Bound<'py, PyArray1<i8>>> {
        let (mx, mn) = (maximo.as_slice()?, minimo.as_slice()?);
        let (c, t) = (cierre.as_slice()?, techo.as_slice()?);
        let nv = nueva.as_slice()?;
        let mut salida = vec![0i8; c.len()];
        let (mut pos, mut pico, mut atr) = (0i8, 0.0f64, 0.0f64);
        for i in 0..c.len() {
            let rango = mx[i] - mn[i];
            atr = if i == 0 { rango }
                  else { alfa * rango + (1.0 - alfa) * atr };
            if i < n || nv[i] {
                pos = 0;
            } else if pos == 0 {
                if c[i] > t[i] { pos = 1; pico = c[i]; }
            } else {
                if c[i] > pico { pico = c[i]; }
                if c[i] < pico - k * atr { pos = 0; }
            }
            salida[i] = pos;
        }
        Ok(salida.into_pyarray(py))
    }
}

# pip/uv: maturin in the dev environment; needs rustc and network
# $ maturin develop --release      (builds and installs into the venv)
from bucle_rust import bucle_rs

pos_rs = bucle_rs(*completo)
assert np.array_equal(pos_rs, pos_nb)

as_slice() fails if the array is not contiguous in memory, instead of reading it wrongly; comparisons with NaN give false just as in Python, so the techo of the first bars behaves the same. The speed would be on the order of Cython and Numba: all three generate machine code for the same loop. The difference lies in everything else:

For a backtest loop, Rust is an outsized tool: you have to learn a new language and maintain a build toolchain. It makes sense when building a library that others will use for years, with its own data structures, threads without the GIL and guarantees that there will be no memory corruption: a backtest engine, a low-latency order manager or a reader for a binary data format.

When to stop

With the loop compiled, the distribution of time has changed. Preparing the arrays (the rolling maximum and the dates, with pandas) takes about 5 milliseconds; the Numba loop, a few tenths. Amdahl's law says what that means:

t_prep = medir(preparar, df, rep=3)
print("preparing now takes longer than the compiled loop:",
      t_prep > t_nb)

def amdahl(p, s):
    return 1 / ((1 - p) + p / s)

for p in (0.5, 0.9, 0.99):
    fila = [f"{amdahl(p, s):7.1f}" for s in (10, 100, 1000)]
    print(f"p = {p:<5}", *fila)

preparing now takes longer than the compiled loop: True
p = 0.5       1.8     2.0     2.0
p = 0.9       5.3     9.2     9.9
p = 0.99      9.2    50.3    91.0

In the table, each row is a fraction $p$ of the time and each column how many times that part is sped up (10, 100 and 1,000). With $p = 0.5$, speeding it up a thousand times gives less than 2. The figure plots it over a continuous range. In the lesson's backtest, the first version spent more than 99% of the time in the loop, and taking pandas out of it changed everything; now the loop is a small fraction and speeding it up further would not change the total. The next step would be to profile the complete program (reading the Parquet file, preparation, analysis of the results) and see whether the total matters. For a backtest of a hundred thousand bars that now takes a few milliseconds, it does not: stop here.

The header figure summarizes the path on the author's machine: each version, with its speedup over the original on the same 20,000 bars. The exact figures change with the machine; the orders of magnitude do not.

Exercises

1. A parameter sweep can be parallelized: each combination is independent. Write a Numba function with parallel=True and prange that, for several values of k, computes the strategy's total log return (previous bar's position times the bar's return) over the 96,330 bars.

from numba import prange

@njit(parallel=True, cache=True)
def barrido(maximo, minimo, cierre, techo, nueva, ks, n=30,
            alfa=2 / 15):
    r = np.zeros(len(ks))
    for j in prange(len(ks)):
        pos = bucle_nb(maximo, minimo, cierre, techo, nueva,
                       n, ks[j], alfa)
        for i in range(1, len(cierre)):
            r[j] += pos[i - 1] * np.log(cierre[i] / cierre[i - 1])
    return r

ks = np.array([1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
for k, r in zip(ks, barrido(*completo, ks)):
    print(f"k = {k:3.0f}: total log return {r:+.4f}")

k =   1: total log return -0.0416
k =   2: total log return +0.0513
k =   3: total log return +0.0703
k =   4: total log return +0.1508
k =   6: total log return +0.1871
k =   8: total log return +0.2855

2. Without touching the logic, remove the column indexing inside the loop of the naive backtest (df["ts"] and df["maximo"]) by moving it before the for, and measure with cProfile how many calls to DataFrame.__getitem__ remain.

def backtest_columnas(df, n=N, k=K, alfa=ALFA):
    ts, maximo = df["ts"], df["maximo"]       # once
    pos, pico, atr = 0, 0.0, None
    salida = []
    for i in range(len(df)):
        fila = df.iloc[i]
        rango = fila["maximo"] - fila["minimo"]
        atr = rango if atr is None else alfa * rango + (1 - alfa) * atr
        if i < n or fila["ts"].date() != ts.iloc[i - 1].date():
            pos = 0
        elif pos == 0:
            if fila["cierre"] > maximo.iloc[i - n:i].max():
                pos, pico = 1, fila["cierre"]
        else:
            pico = max(pico, fila["cierre"])
            if fila["cierre"] < pico - k * atr:
                pos = 0
        salida.append(pos)
    return np.array(salida, dtype=np.int8)

perfil = cProfile.Profile()
pos5 = perfil.runcall(backtest_columnas, df.iloc[:5_000])
st = pstats.Stats(perfil)
llamadas = {f[2]: v[1] for f, v in st.stats.items()
            if f[0].endswith("frame.py")}
print("DataFrame.__getitem__:", llamadas.get("__getitem__"))
print("same result:",
      np.array_equal(pos5, backtest_ingenuo(df.iloc[:5_000])))

DataFrame.__getitem__: 2
same result: True

3. A colleague wants to rewrite in Rust the loop of their strategy, which takes 2 seconds inside a 40-minute nightly retraining process. What would you tell them?