---
title: "7.1.2 Performance: profiling, Numba, Cython and Rust"
description: "Profile a slow backtest with cProfile, pyinstrument and py-spy, take pandas out of the loop, compile it with Numba and Cython; 3.15 sampling and Rust."
url: "https://pythonparatrading.com/en/python-trading-course/level-7/python-performance-profiling-numba-cython-rust/"
lang: "en"
image: "https://pythonparatrading.com/og/en/python-trading-course/level-7/python-performance-profiling-numba-cython-rust.png"
---
# 7.1.2 · Performance: measure, profile and compile only what is needed
> A breakout backtest with a trailing stop takes almost twenty seconds to go through a hundred thousand one-minute bars. This lesson measures it, uses three profilers to find where the time goes (it is not where it seems), rewrites it without pandas inside the loop and compiles it with Numba and with Cython until the loop stops mattering. Then it introduces Python 3.15's sampling profiler and the Rust route with PyO3.
- Course: [Level 7: Production, MLOps, regulation, career and capstone](https://pythonparatrading.com/en/python-trading-course/level-7/)
- Module: 7.1 · Software engineering for trading
- Duration: 75 min
- Prerequisites: [0.4.6 · Concurrency: threading, multiprocessing, asyncio and free-threading](https://pythonparatrading.com/en/python-trading-course/level-0/python-concurrency-threading-multiprocessing-asyncio/), [0.5.1 · NumPy 2: arrays, broadcasting, vectorization, linear algebra and random generators](https://pythonparatrading.com/en/python-trading-course/level-0/numpy-arrays-and-vectorization/), [0.5.2 · pandas 3.0: Series, DataFrame, time indexes, Copy-on-Write, the str type and pd.col()](https://pythonparatrading.com/en/python-trading-course/level-0/pandas-3-series-and-dataframe/), [2.4.2 · Event-driven backtest from scratch: loop, portfolio, orders and fills](https://pythonparatrading.com/en/python-trading-course/level-2/event-driven-backtest-from-scratch/), [7.1.1 · Project structure, configuration and quality: from notebook to repository](https://pythonparatrading.com/en/python-trading-course/level-7/python-project-structure-configuration-quality/)
## By the end of this lesson
- Measure before optimizing and use Amdahl's law to know the most that speeding up one part of the program can gain.
- Read a deterministic profile (cProfile) and a sampling one (pyinstrument, py-spy) and explain how they differ, including Python 3.15's sampling profiler.
- Rewrite a path-dependent loop without pandas objects inside it and check that it gives exactly the same result.
- Compile that loop with Numba and with Cython, read Cython's annotation and know what limits each tool.
- Place Rust with PyO3 and maturin relative to Numba and Cython: what it offers and what it costs.
## Measure before optimizing
The slow code in a trading system is almost never the strategy's arithmetic. It is usually the way the data are
traversed: one pandas row at a time, a conversion on every iteration, a disk read inside a loop. Guessing where the
problem is, is the most common way to lose an afternoon optimizing what does not matter. The method is always the
same:
```mermaid
flowchart TB
M[Measure] --> P[Profile]
P --> A[Algorithm]
A --> C[Compile]
C --> M2[Measure again]
```
First measure the whole program with realistic data; then profile it to find out which part takes the time; change the
algorithm or the data structure (almost always what pays off most) and, only if that is still not enough, compile the
hot part. Then measure again, because the bottleneck moves. How much can be gained is bounded by Amdahl's law: if a
part takes the fraction $p$ of the time and is sped up $s$ times, the whole program is sped up
$$
S = \frac{1}{(1 - p) + \dfrac{p}{s}}
$$
With $p = 0.9$, speeding that part up 100 times gives $S \approx 9.2$; not even with an infinite $s$ does it exceed 10.
The figure at the end of the lesson plots the curve for several values of $p$.
The lesson's case is an intraday breakout strategy on the 96,330 one-minute bars of the level 5 synthetic future
(`hf5_barras_1min.parquet`): it buys when the close exceeds the high of the previous 30 bars; it exits when the close
falls more than 4 ATR below the highest close reached since the entry (a trailing stop) or when a new session starts.
The ATR here is a 14-bar exponential average of each bar's range (high minus low). The exit depends on the path (the
highest close since the entry, whether the position is open or not), so it cannot be written with column operations:
it needs a loop, as in lesson 2.4.2. This is how someone coming from a notebook would write it:
```python
import time
import numpy as np
import pandas as pd
df = pd.read_parquet("hf5_barras_1min.parquet")
N, K, ALFA = 30, 4.0, 2 / 15 # 30-bar breakout, 4-ATR stop
def backtest_ingenuo(df, n=N, k=K, alfa=ALFA):
"""Position (0 or 1) on each bar."""
pos, pico, atr = 0, 0.0, None
salida = []
for i in range(len(df)):
fila = df.iloc[i]
rango = fila["maximo"] - fila["minimo"]
atr = rango if atr is None else alfa * rango + (1 - alfa) * atr
if i < n or fila["ts"].date() != df["ts"].iloc[i - 1].date():
pos = 0 # new session
elif pos == 0:
techo = df["maximo"].iloc[i - n:i].max()
if fila["cierre"] > techo:
pos, pico = 1, fila["cierre"] # entry
else:
pico = max(pico, fila["cierre"])
if fila["cierre"] < pico - k * atr:
pos = 0 # trailing stop
salida.append(pos)
return np.array(salida, dtype=np.int8)
sub = df.iloc[:20_000]
t0 = time.perf_counter()
pos_ref = backtest_ingenuo(sub)
t_ingenuo = time.perf_counter() - t0 # not printed
entradas = int((np.diff(pos_ref, prepend=0) == 1).sum())
print(len(df), "bars,", df["ts"].dt.date.nunique(), "sessions")
print(f"first 20,000: {entradas} entries, "
f"in the market {pos_ref.mean():.1%} of the bars")
```
```output
96330 bars, 247 sessions
first 20,000: 226 entries, in the market 40.7% of the bars
```
On the machine the lesson was written on, the first 20,000 bars take about 3.7 seconds: about 185 microseconds per
bar, about 18 seconds for the whole file. The cells do not print timings because they change from one machine to
another and from one run to another; they store the time in a variable and, when comparing, print only whether one
version is faster than another by a wide margin. For a backtest that runs once, 18 seconds do not matter. For a sweep of
200 parameter combinations (lesson 2.5.6), or for a hundred assets, they are an hour.
## cProfile: exact counts
`cProfile` is the standard library's deterministic profiler: it intercepts every function call and return and records
how many times each one was called and how long it took. Its default report (`pstats`) is long; what matters here are
the functions that `backtest_ingenuo` calls directly, with their number of calls, sorted by cumulative time. 5,000
bars are enough:
```python
import cProfile
import pstats
perfil = cProfile.Profile()
perfil.runcall(backtest_ingenuo, df.iloc[:5_000])
st = pstats.Stats(perfil)
raiz = next(f for f in st.stats if f[2] == "backtest_ingenuo")
total = st.stats[raiz][3] # cumulative time
hijos = []
for (fichero, _, nombre), (*_, llamadores) in st.stats.items():
if raiz in llamadores:
_, ncalls, _, acumulado = llamadores[raiz]
corto = fichero.rsplit("site-packages/", 1)[-1]
hijos.append((acumulado, ncalls, f"{corto}:{nombre}"))
hijos.sort(reverse=True)
for acumulado, ncalls, nombre in hijos[:2]:
print(f"{ncalls:>7} {nombre}")
pandas = sum(a for a, _, n in hijos if n.startswith("pandas"))
print("pandas, more than 80% of the time:", pandas / total > 0.8)
```
```output
13038 pandas/core/indexing.py:__getitem__
8038 pandas/core/frame.py:__getitem__
pandas, more than 80% of the time: True
```
The two functions that accumulate the most time are pandas' indexing functions: `iloc[...]` and `df[...]`. The counts
can be reconstructed by reading the code, and it is worth doing: the 13,038 calls to `_iLocIndexer.__getitem__` are the
5,000 from `df.iloc[i]`, the 4,970 from `df["ts"].iloc[i - 1]` (every bar except the first 30) and 3,068 from
`df["maximo"].iloc[i - n:i]` (only when out of the market); the 8,038 calls to `DataFrame.__getitem__` are the
`df["ts"]` and `df["maximo"]` of the last two. On the author's machine, `iloc[...]` takes more than half of the time,
`df[...]` more than a quarter, and the strategy's arithmetic (the function's `[self]` in the pyinstrument report
below) around 4%.
cProfile has two drawbacks worth knowing. It adds a fixed cost to every Python call, so it distorts code with many
small calls (like this one) more; the percentages are indicative and the absolute times higher than without
profiling. And it only measures the thread it is enabled from. What it gives in exchange are exact counts: if a
function is called a million times when it should be called a thousand, cProfile says so without ambiguity.
## Sampling profilers: pyinstrument and py-spy
A sampling profiler does not intercept calls: it looks at the program's stack every so often (every millisecond, for
example) and counts which functions it was in. A function that appears in 50% of the samples took, approximately, half
of the time. It loses the exact counts and very short functions, but it interferes much less with the program.
pyinstrument samples from inside the process and presents a call tree. Its text output carries the time of day, the
number of samples and the seconds of each branch, all variable; the cell walks the tree and writes only the branches
under `backtest_ingenuo` that exceed 20% of the time:
```python
from pyinstrument import Profiler
from pyinstrument.renderers import ConsoleRenderer
pf = Profiler(interval=0.001)
pf.start()
backtest_ingenuo(df.iloc[:5_000])
pf.stop()
arbol = ConsoleRenderer().preprocess(pf.last_session.root_frame())
def buscar(marco, nombre):
if marco.function == nombre:
return marco
for hijo in marco.children:
if (r := buscar(hijo, nombre)) is not None:
return r
b = buscar(arbol, "backtest_ingenuo")
for hijo in b.children:
if hijo.time / b.time > 0.2:
nombre = f"{hijo.class_name}.{hijo.function}"
print(nombre, hijo.file_path_short)
```
```output
_iLocIndexer.__getitem__ pandas/core/indexing.py
DataFrame.__getitem__ pandas/core/frame.py
```
The same two branches as cProfile. pyinstrument's full report, as printed by `output_text()` when running this same
code as a script (`pyi.py`) on the author's machine (the profile's path shortened; the timings change on every run),
shows the grouped tree, with pandas' internal frames collapsed:
```python skip
print(pf.output_text(unicode=True, color=False))
```
```output
_ ._ __/__ _ _ _ _ _/_ Recorded: 23:43:42 Samples: 1774
/_//_/// /_\ / //_// / //_'/ // Duration: 1.774 CPU time: 1.753
/ _/ v5.1.3
Profile at pyi.py:33
1.773 pyi.py:1
└─ 1.773 backtest_ingenuo pyi.py:9
├─ 0.962 _iLocIndexer.__getitem__ pandas/core/indexing.py:1192
│ [38 frames hidden] pandas, typing, annotationlib,
├─ 0.112 Series.__getitem__ pandas/core/series.py:945
│ [5 frames hidden] pandas
└─ 0.067 [self] pyi.py
```
py-spy samples from **outside** the process: it is a separate program (written in Rust) that reads the interpreter's
memory. That is why the code does not have to be touched and it works on a bot that is already running. On Linux,
reading another process's memory requires permissions: with `kernel.yama.ptrace_scope = 1`, the usual setting, only a
child process can be sampled, so py-spy launches the script itself. The cell writes a script with the same backtest
(extracted with `inspect.getsource`), runs it under py-spy at 200 samples per second and counts, in the `raw` format
(one line per distinct stack: the frames separated by `;` and the number of samples), which of the script's functions
is deepest in the stack in each sample, not counting the module level (``):
```python
import inspect
import subprocess
from collections import Counter
from pathlib import Path
Path("perfilar.py").write_text(
"import numpy as np\nimport pandas as pd\n\n"
"N, K, ALFA = 30, 4.0, 2 / 15\n\n"
+ inspect.getsource(backtest_ingenuo)
+ "\n\ndef cargar():\n"
" return pd.read_parquet('hf5_barras_1min.parquet')\n\n"
"backtest_ingenuo(cargar().iloc[:5_000])\n")
r = subprocess.run(["py-spy", "record", "--nonblocking", "-r", "200",
"-f", "raw", "-o", "perfil.txt", "--",
"python", "perfilar.py"],
capture_output=True, text=True)
cuenta = Counter()
for linea in Path("perfil.txt").read_text().splitlines():
pila, n = linea.rsplit(" ", 1)
propios = [m.split(" (")[0] for m in pila.split(";")
if "(perfilar.py:" in m and "" not in m]
if propios:
cuenta[propios[-1]] += int(n)
print("py-spy:", r.returncode, "| most sampled own function:",
cuenta.most_common(1)[0][0])
```
```output
py-spy: 0 | most sampled own function: backtest_ingenuo
```
With Python 3.14 and py-spy 0.4.2, on this machine, many `--nonblocking` samples come out empty or truncated: the
process keeps running while py-spy reads its stack, and sometimes it reads it halfway. That is why the cell discards the
module level, where the truncated stacks end up (along with the pandas import), and compares only the script's
functions. The blocking mode, which stops the process at each sample, is reliable but slows this script down dozens of
times. For coarse questions (which of my functions takes the time) it is enough; for fine percentages, pyinstrument or
cProfile are better.
## Python 3.15's sampling profiler
Python 3.15 (released on October 9, 2026[^1]) reorganizes the profilers into a new package, `profiling` (PEP
799[^2]): `profiling.tracing` is the usual cProfile under another name, the old `profile` module (in pure Python) is
deprecated, and `profiling.sampling`, nicknamed Tachyon, is a sampling profiler like py-spy but included in the
interpreter[^3]. According to its documentation:
- It reads the stack from outside the process, without stopping it, at a cost to the program that the documentation
describes as virtually zero. By default it takes 1,000 samples per second (`-r`); the duration can be limited (`-d`).
- It has four modes: wall-clock time (`--mode=wall`, the default, includes network and disk waits), CPU
(`--mode=cpu`, only when the thread is executing), GIL (`--mode=gil`, only when it holds the GIL: separates Python
from C extensions) and exceptions.
- It can launch a script (`run`), attach to a process by its PID (`attach`) or read a single snapshot of the stack
(`dump`, useful with a hung process), and it writes pstats, collapsed stacks (the same format as py-spy's `raw`),
HTML flame graphs, per-line heat maps or the Firefox Profiler format.
- On Linux, `attach` and `dump` have the same `ptrace_scope` restriction as py-spy; the profiler and the process must
be the same minor Python version (and the same exact version if either is a pre-release).
The course environment is Python 3.14 (uv, on this machine, only offers 3.15.0rc2), so the following code is not run.
With Python 3.15, the py-spy cell can be rewritten without installing anything, and the count is the same because the
collapsed format is the same:
```python skip
# Python 3.15: standard library sampling
r = subprocess.run([sys.executable, "-m", "profiling.sampling",
"run", "--collapsed", "-r", "10khz",
"perfilar.py"], capture_output=True, text=True)
# one line per stack: "frame;frame;frame N", like py-spy's raw
# A running bot (same user, ptrace permissions):
# python -m profiling.sampling attach --mode=cpu -d 30
# python -m profiling.sampling dump # where is it stuck?
```
For a live system (lesson 4.5.1), `dump` answers a frequent question at three in the morning: “what is the process
that is not responding doing right now?”. Until now that required py-spy or gdb.
## First, take pandas out of the loop
The profile says the problem is asking pandas for a row and a slice of a column on every iteration. Whatever does not
depend on the path is computed outside, with column operations: the high of the previous 30 bars is a rolling maximum
shifted by one bar, and the session change is comparing the date with the previous bar's. The loop receives only
NumPy arrays:
```mermaid
flowchart LR
P[prepare: pandas] --> A[NumPy arrays]
A --> B[compilable loop]
B --> S[positions]
```
```python
def preparar(df, n=N):
techo = df["maximo"].rolling(n).max().shift().to_numpy()
dia = df["ts"].dt.floor("D")
nueva = (dia != dia.shift()).to_numpy()
return (df["maximo"].to_numpy(), df["minimo"].to_numpy(),
df["cierre"].to_numpy(), techo, nueva)
def bucle(maximo, minimo, cierre, techo, nueva, n=N, k=K, alfa=ALFA):
m = len(cierre)
salida = np.zeros(m, dtype=np.int8)
pos, pico, atr = 0, 0.0, 0.0
for i in range(m):
rango = maximo[i] - minimo[i]
atr = rango if i == 0 else alfa * rango + (1 - alfa) * atr
if i < n or nueva[i]:
pos = 0
elif pos == 0:
if cierre[i] > techo[i]:
pos, pico = 1, cierre[i]
else:
if cierre[i] > pico:
pico = cierre[i]
if cierre[i] < pico - k * atr:
pos = 0
salida[i] = pos
return salida
import timeit
def medir(f, *args, veces=1, rep=5):
"""Best time per call over `rep` repetitions."""
return min(timeit.repeat(lambda: f(*args), number=veces,
repeat=rep)) / veces
arrays = preparar(sub)
print("identical to the naive version:",
np.array_equal(bucle(*arrays), pos_ref))
t_arrays = medir(bucle, *arrays)
print("more than 50 times faster:", t_ingenuo / t_arrays > 50)
```
```output
identical to the naive version: True
more than 50 times faster: True
```
Same result, bar by bar, and more than 50 times faster (on the author's machine, about 280 times). The comparison with
the reference result is the important part: an optimization that changes a single position is not an optimization, it
is a different backtest. Two equivalence details: the `techo` shifted by one bar reproduces `iloc[i - n:i]`, which does
not include the current bar, and over the first 30 bars it is NaN, which is never less than a close, just as in the
original version, which does not evaluate them.
The loop is still Python: each `maximo[i]` creates a Python `float` object from the array, each addition creates
another. That is why, curiously, the same loop with Python lists (`arr.tolist()`) runs somewhat faster still than with
arrays: NumPy is fast with operations on whole arrays, not with individual elements. To go further you have to leave
the interpreter.
## Numba: the same loop, compiled
Numba compiles Python functions to machine code with LLVM at the time of the first call, based on the argument
types[^4]. In `nopython` mode (the one used by `@njit`) the result does not use the interpreter at all: numbers are
machine numbers and arrays, contiguous memory. The `bucle` function does not have to be rewritten:
```python
from numba import njit
bucle_nb = njit(cache=True)(bucle)
completo = preparar(df)
pos_nb = bucle_nb(*completo) # compiles here
pos_py = bucle(*completo)
print("identical to the Python loop on the 96,330 bars:",
np.array_equal(pos_nb, pos_py))
t_py = medir(bucle, *completo, rep=3)
t_nb = medir(bucle_nb, *completo, veces=20)
print("Numba, more than 20 times faster than the Python loop:",
t_py / t_nb > 20)
```
```output
identical to the Python loop on the 96,330 bars: True
Numba, more than 20 times faster than the Python loop: True
```
On the author's machine, about 240 times: the loop over the 96,330 bars goes from about 58 milliseconds to about 0.24.
The first call compiles (around half a second for a function like this) and `cache=True` stores the result on disk,
so that the next process does not compile again as long as neither the code nor the types change. That is why the cell
measures after the first call.
What Numba cannot compile fails with a typing error, not with a slow program. Passing it the DataFrame instead of the
arrays:
```python
from numba.core.errors import TypingError
ingenuo_nb = njit(backtest_ingenuo)
try:
ingenuo_nb(df.iloc[:100])
except TypingError as e:
print(type(e).__name__ + ":", str(e).splitlines()[0])
```
```output
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
```
Numba understands numbers, NumPy arrays, tuples, some typed dictionaries and lists and a large part of NumPy; it does
not understand pandas or arbitrary objects. That shapes the design: the boundary between pandas and the compiled code
is made of arrays, as in `preparar`. Three more precautions. `fastmath=True` speeds up some operations by allowing
floating-point sums to be reordered, which changes the last decimals: in a backtest that is compared against a
reference result, better not to use it. `parallel=True` with `prange` distributes independent iterations across
threads, and this loop has none (each bar depends on the previous one), but a parameter sweep does (exercise 1). And
compiled code does not check out-of-range indices by default: an error that in Python would raise `IndexError` can read
someone else's memory without warning.
## Cython: declared types and C
Cython translates a dialect of Python with C types into a C extension compiled with the system compiler (here, gcc).
Without type declarations, the translated code still calls the Python API on every operation and gains little; with
types (`cdef double`, `double[:]` memoryviews over the arrays), the loop becomes pure C[^5]. The cell writes the typed
version to a `.pyx` file, annotates it with `cython -a` alongside an untyped copy of `bucle` and counts, in each
annotation, how many lines of the loop body call the Python API (the ones Cython highlights in yellow in the HTML):
```python
import re
import sys
PYX = '''# cython: boundscheck=False, wraparound=False
import numpy as np
def bucle_cy(const double[:] maximo, const double[:] minimo,
const double[:] cierre, const double[:] techo,
const unsigned char[:] nueva, int n=30,
double k=4.0, double alfa=2.0 / 15):
cdef Py_ssize_t i, m = cierre.shape[0]
cdef double rango, pico = 0.0, atr = 0.0
cdef signed char pos = 0
salida = np.zeros(m, dtype=np.int8)
cdef signed char[:] s = salida
for i in range(m):
rango = maximo[i] - minimo[i]
atr = rango if i == 0 else alfa * rango + (1 - alfa) * atr
if i < n or nueva[i]:
pos = 0
elif pos == 0:
if cierre[i] > techo[i]:
pos = 1
pico = cierre[i]
else:
if cierre[i] > pico:
pico = cierre[i]
if cierre[i] < pico - k * atr:
pos = 0
s[i] = pos
return salida
'''
Path("bucle_cy.pyx").write_text(PYX)
Path("bucle_sin_tipos.pyx").write_text(
"import numpy as np\n\n" + inspect.getsource(bucle)
.replace("n=N, k=K, alfa=ALFA", "n=30, k=4.0, alfa=2 / 15"))
def lineas_con_python(nombre):
"""Loop-body lines with calls to the Python API."""
subprocess.run(["cython", "-3", "-a", f"{nombre}.pyx"], check=True)
html = Path(f"{nombre}.html").read_text()
puntos = [int(p) for p in
re.findall(r'class="cython line score-(\d+)"', html)]
codigo = Path(f"{nombre}.pyx").read_text().splitlines()
ini = next(i for i, l in enumerate(codigo) if "for i in" in l)
fin = next(i for i, l in enumerate(codigo) if "return" in l)
return sum(p > 0 for p in puntos[ini + 1:fin]), fin - ini - 1
for nombre in ("bucle_sin_tipos", "bucle_cy"):
con, total = lineas_con_python(nombre)
print(f"{nombre}: {con} of {total} loop lines use Python")
```
```output
bucle_sin_tipos: 9 of 13 loop lines use Python
bucle_cy: 0 of 14 loop lines use Python
```
Without types, nine of the loop's thirteen lines go through Python objects; with types, none. The annotation is the
working tool with Cython: you write, annotate, look at the yellow lines inside the loops and add types until they
disappear. The directives on the first line remove the index checks (`boundscheck`) and the negative-index handling
(`wraparound`): they are faster and as dangerous as in Numba. `pyximport` compiles the `.pyx` when it is imported, in a
temporary folder:
```python
import pyximport
sys.path.insert(0, ".")
pyximport.install(build_dir="build_pyx", language_level=3)
from bucle_cy import bucle_cy
maximo, minimo, cierre, techo, nueva = completo
pos_cy = bucle_cy(maximo, minimo, cierre, techo, nueva.view(np.uint8))
print("identical to Numba:", np.array_equal(pos_cy, pos_nb))
t_cy = medir(bucle_cy, maximo, minimo, cierre, techo,
nueva.view(np.uint8), veces=20)
print("Cython, same order as Numba (between 0.2 and 5 times):",
0.2 < t_nb / t_cy < 5)
```
```output
identical to Numba: True
Cython, same order as Numba (between 0.2 and 5 times): True
```
Same result and the same order of magnitude as Numba. On the author's machine, with the whole file, Cython takes about
0.19 milliseconds and Numba about 0.24; with 20,000 bars, Cython comes out about 2.7 times faster. The differences
between two compiled versions depend on the size and on details such as the fixed cost of each call; what matters is
that both are hundreds of times ahead of the Python loop. `nueva.view(np.uint8)` reinterprets the boolean array as
bytes without copying it, because Cython memoryviews do not accept `bool` directly. To distribute a package with Cython
you do not use `pyximport` but the package's build system: a `setup.py` with `cythonize` or a backend such as
`setuptools` declared in `pyproject.toml`, which compiles at install time and produces wheels per platform.
Numba or Cython? For a numerical research loop, Numba: it is Python, there is no separate compilation step and no
compiler is needed on the machine. Cython is the better fit when you need to call C libraries, mix compiled code with
Python objects or distribute an extension without the user installing LLVM; it is what pandas, scikit-learn or
NautilusTrader use in parts of their core (lesson 2.4.5).
## Rust with PyO3: what you need to know
Rust is a compiled language with memory safety guaranteed by the compiler: no Cython-style `boundscheck=False` and no
silent out-of-range accesses (in Rust, an out-of-range index stops the program with an error). PyO3 generates the
bindings to Python and maturin builds and publishes the wheels[^6]. Half of the quantitative Python ecosystem's stack is
already written this way: Polars (lesson 0.5.4), the NautilusTrader core, Ruff, uv and py-spy itself.
The same loop as a Rust extension. Compiling it requires downloading crates (Rust dependencies) from crates.io, and the
course does not use the network, so the code is for reading only, with the versions published as of October 23, 2026
(pyo3 0.30.1, numpy 0.30.0):
```toml
# Cargo.toml
[package]
name = "bucle_rust"
version = "0.1.0"
edition = "2024"
[lib]
name = "bucle_rust"
crate-type = ["cdylib"]
[dependencies]
pyo3 = "0.30"
numpy = "0.30"
```
```rust
// src/lib.rs
#[pyo3::pymodule]
mod bucle_rust {
use numpy::{IntoPyArray, PyArray1, PyReadonlyArray1};
use pyo3::prelude::*;
#[pyfunction]
#[pyo3(signature = (maximo, minimo, cierre, techo, nueva,
n=30, k=4.0, alfa=2.0 / 15.0))]
fn bucle_rs<'py>(
py: Python<'py>,
maximo: PyReadonlyArray1<'py, f64>,
minimo: PyReadonlyArray1<'py, f64>,
cierre: PyReadonlyArray1<'py, f64>,
techo: PyReadonlyArray1<'py, f64>,
nueva: PyReadonlyArray1<'py, bool>,
n: usize, k: f64, alfa: f64,
) -> PyResult>> {
let (mx, mn) = (maximo.as_slice()?, minimo.as_slice()?);
let (c, t) = (cierre.as_slice()?, techo.as_slice()?);
let nv = nueva.as_slice()?;
let mut salida = vec![0i8; c.len()];
let (mut pos, mut pico, mut atr) = (0i8, 0.0f64, 0.0f64);
for i in 0..c.len() {
let rango = mx[i] - mn[i];
atr = if i == 0 { rango }
else { alfa * rango + (1.0 - alfa) * atr };
if i < n || nv[i] {
pos = 0;
} else if pos == 0 {
if c[i] > t[i] { pos = 1; pico = c[i]; }
} else {
if c[i] > pico { pico = c[i]; }
if c[i] < pico - k * atr { pos = 0; }
}
salida[i] = pos;
}
Ok(salida.into_pyarray(py))
}
}
```
```python skip
# pip/uv: maturin in the dev environment; needs rustc and network
# $ maturin develop --release (builds and installs into the venv)
from bucle_rust import bucle_rs
pos_rs = bucle_rs(*completo)
assert np.array_equal(pos_rs, pos_nb)
```
`as_slice()` fails if the array is not contiguous in memory, instead of reading it wrongly; comparisons with NaN give
`false` just as in Python, so the `techo` of the first bars behaves the same. The speed would be on the order of
Cython and Numba: all three generate machine code for the same loop. The difference lies in everything else:
| | Numba | Cython | Rust (PyO3) |
|---|---|---|---|
| Language | Python (subset) | Python with C types | Rust |
| Compilation | at run time (LLVM) | at install time (C compiler) | at install time (rustc) |
| Memory safety | no index checking by default | depends on the directives | guaranteed by the compiler |
| Distribution | depends on Numba and llvmlite | wheel per platform | wheel per platform (maturin) |
| Suited for | numerical research loops | extensions that mix C and Python | large, concurrent libraries |
For a backtest loop, Rust is an outsized tool: you have to learn a new language and maintain a build toolchain. It
makes sense when building a library that others will use for years, with its own data structures, threads without the
GIL and guarantees that there will be no memory corruption: a backtest engine, a low-latency order manager or a reader
for a binary data format.
## When to stop
With the loop compiled, the distribution of time has changed. Preparing the arrays (the rolling maximum and the dates,
with pandas) takes about 5 milliseconds; the Numba loop, a few tenths. Amdahl's law says what that means:
```python
t_prep = medir(preparar, df, rep=3)
print("preparing now takes longer than the compiled loop:",
t_prep > t_nb)
def amdahl(p, s):
return 1 / ((1 - p) + p / s)
for p in (0.5, 0.9, 0.99):
fila = [f"{amdahl(p, s):7.1f}" for s in (10, 100, 1000)]
print(f"p = {p:<5}", *fila)
```
```output
preparing now takes longer than the compiled loop: True
p = 0.5 1.8 2.0 2.0
p = 0.9 5.3 9.2 9.9
p = 0.99 9.2 50.3 91.0
```
*Fig. — Amdahl's law: whole-program speed-up as a function of how much the optimised part is sped up, for three time fractions p. With p = 0.9, a hundredfold speed-up gives 9.2 and the ceiling is 10.* Three curves with a logarithmic horizontal axis from 1 to 1,000. With p = 0.5 the curve flattens quickly near 2. With p = 0.9 it rises to almost 10; the highlighted point, s = 100, gives S = 9.2. With p = 0.99 it rises to about 91 at s = 1,000.
In the table, each row is a fraction $p$ of the time and each column how many times that part is sped up (10, 100 and
1,000). With $p = 0.5$, speeding it up a thousand times gives less than 2. The figure plots it over a continuous range.
In the lesson's backtest, the first version spent more than 99% of the time in the loop, and taking pandas out of it
changed everything; now the loop is a small fraction and speeding it up further would not change the total. The next
step would be to profile the complete program (reading the Parquet file, preparation, analysis of the results) and see
whether the total matters. For a backtest of a hundred thousand bars that now takes a few milliseconds, it does not:
stop here.
The header figure summarizes the path on the author's machine: each version, with its speedup over the original on the
same 20,000 bars. The exact figures change with the machine; the orders of magnitude do not.
## Exercises
1. A parameter sweep can be parallelized: each combination is independent. Write a Numba function with
`parallel=True` and `prange` that, for several values of `k`, computes the strategy's total log return (previous
bar's position times the bar's return) over the 96,330 bars.
**Solution.**
```python
from numba import prange
@njit(parallel=True, cache=True)
def barrido(maximo, minimo, cierre, techo, nueva, ks, n=30,
alfa=2 / 15):
r = np.zeros(len(ks))
for j in prange(len(ks)):
pos = bucle_nb(maximo, minimo, cierre, techo, nueva,
n, ks[j], alfa)
for i in range(1, len(cierre)):
r[j] += pos[i - 1] * np.log(cierre[i] / cierre[i - 1])
return r
ks = np.array([1.0, 2.0, 3.0, 4.0, 6.0, 8.0])
for k, r in zip(ks, barrido(*completo, ks)):
print(f"k = {k:3.0f}: total log return {r:+.4f}")
```
```output
k = 1: total log return -0.0416
k = 2: total log return +0.0513
k = 3: total log return +0.0703
k = 4: total log return +0.1508
k = 6: total log return +0.1871
k = 8: total log return +0.2855
```
A Numba function can call another one (`bucle_nb`) inside the compiled code. Each `k` writes only to its own position
in `r`, so there are no race conditions, and the sum for each `k` is done in a single thread and in order: the result
is the same with one thread or with four. The figures come from synthetic data and say nothing about the strategy, and
choosing `k` by the maximum of this table would be overfitting (lesson 2.5.4).
2. Without touching the logic, remove the column indexing inside the loop of the naive backtest (`df["ts"]` and
`df["maximo"]`) by moving it before the `for`, and measure with cProfile how many calls to `DataFrame.__getitem__`
remain.
**Solution.**
```python
def backtest_columnas(df, n=N, k=K, alfa=ALFA):
ts, maximo = df["ts"], df["maximo"] # once
pos, pico, atr = 0, 0.0, None
salida = []
for i in range(len(df)):
fila = df.iloc[i]
rango = fila["maximo"] - fila["minimo"]
atr = rango if atr is None else alfa * rango + (1 - alfa) * atr
if i < n or fila["ts"].date() != ts.iloc[i - 1].date():
pos = 0
elif pos == 0:
if fila["cierre"] > maximo.iloc[i - n:i].max():
pos, pico = 1, fila["cierre"]
else:
pico = max(pico, fila["cierre"])
if fila["cierre"] < pico - k * atr:
pos = 0
salida.append(pos)
return np.array(salida, dtype=np.int8)
perfil = cProfile.Profile()
pos5 = perfil.runcall(backtest_columnas, df.iloc[:5_000])
st = pstats.Stats(perfil)
llamadas = {f[2]: v[1] for f, v in st.stats.items()
if f[0].endswith("frame.py")}
print("DataFrame.__getitem__:", llamadas.get("__getitem__"))
print("same result:",
np.array_equal(pos5, backtest_ingenuo(df.iloc[:5_000])))
```
```output
DataFrame.__getitem__: 2
same result: True
```
2 calls remain out of the 8,038: the two before the loop. The result does not change and the backtest saves what
cProfile attributed to `DataFrame.__getitem__`, more than a quarter of the time on the author's machine. It is the
first step of the lesson's rewrite, done with the profile in hand: take out of the loop whatever does not depend on
the iteration.
3. A colleague wants to rewrite in Rust the loop of their strategy, which takes 2 seconds inside a 40-minute nightly
retraining process. What would you tell them?
**Solution.**
To measure first. By Amdahl's law, the loop is 2/2,400 ≈ 0.08% of the process: even if it took zero, retraining would
still last 40 minutes minus 2 seconds. The first thing is to profile the whole process (with pyinstrument, py-spy or,
with Python 3.15, `profiling.sampling`) to find out where the 40 minutes go. If that loop really needed speeding up,
Numba would do it in one line without changing language; Rust would make sense if the loop were part of a library that
is going to be maintained for years, with several threads or its own data structures.
[^1]: Python.org, “Python 3.15.0” (https://www.python.org/downloads/release/python-3150/): release date, October 9,
2026. Accessed on October 23, 2026.
[^2]: PEP 799, “A dedicated `profiling` package for organizing Python profiling tools”
(https://peps.python.org/pep-0799/). Accessed on October 23, 2026.
[^3]: Python 3.15 documentation, “profiling.sampling — Statistical profiler”
(https://docs.python.org/3.15/library/profiling.sampling.html): modes, default sampling rate, the `run`, `attach`,
`dump` and `replay` commands, output formats and platform requirements. Accessed on October 23, 2026.
[^4]: Numba, “Compiling Python code with @jit” (https://numba.readthedocs.io/en/stable/user/jit.html). Accessed on
October 23, 2026.
[^5]: Cython, “Typed Memoryviews” (https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html). Accessed on
October 23, 2026.
[^6]: PyO3 (https://pyo3.rs/) and its README, with the `Cargo.toml` example and `#[pyo3::pymodule]` for version 0.30;
rust-numpy (https://github.com/PyO3/rust-numpy) for `PyReadonlyArray1` and `IntoPyArray`; versions on crates.io.
Accessed on October 23, 2026.
## Self-assessment
1. 90% of a program's time is spent in one function. You make it 100 times faster. How much faster does the whole program run, according to Amdahl's law?
- A. 100 times
- B. 90 times
- C. About 9.2 times: the 10% that was not touched comes to dominate
- D. About 50 times
*Answer C: About 9.2 times: the 10% that was not touched comes to dominate.* S = 1 / ((1 − p) + p/s) = 1 / (0.1 + 0.9/100) ≈ 9.17. Even if the function were infinitely fast, the program would not exceed 10 times. That is why you first measure what fraction the part you want to speed up takes.
2. cProfile counts 13,038 calls to `_iLocIndexer.__getitem__` when going through 5,000 bars. Where do they come from?
- A. They are an artifact of the profiler
- B. From `df.iloc[i]` (5,000), `df["ts"].iloc[i - 1]` (4,970) and `df["maximo"].iloc[i - n:i]` (3,068): each bar pays for two or three pandas indexing operations
- C. From `np.array` at the end
- D. From reading the Parquet file
*Answer B: From `df.iloc[i]` (5,000), `df["ts"].iloc[i - 1]` (4,970) and `df["maximo"].iloc[i - n:i]` (3,068): each bar pays for two or three pandas indexing operations.* A deterministic profiler counts every call exactly, and the counts can be reconstructed from the code. That is the diagnosis: the time does not go into the strategy's arithmetic but into asking pandas for a row and a slice of a column on every iteration.
3. Why can't the Numba version receive the DataFrame directly?
- A. Because Numba compiles in nopython mode: it only understands types it knows how to translate into machine code (numbers, NumPy arrays, tuples…), not pandas objects
- B. Because Numba only works on GPUs
- C. Because Numba does not support loops
- D. Because the DataFrame is too large
*Answer A: Because Numba compiles in nopython mode: it only understands types it knows how to translate into machine code (numbers, NumPy arrays, tuples…), not pandas objects.* Numba infers the argument types and generates code with LLVM; a DataFrame is a Python object it cannot type, and compilation fails with `TypingError`. The columns are passed as arrays with `.to_numpy()`.
4. What advantage does a sampling profiler (pyinstrument, py-spy, Python 3.15's `profiling.sampling`) have over a deterministic one like cProfile?
- A. It gives the exact number of calls
- B. It is more accurate on very short functions
- C. It only works with Python 3.15
- D. It interferes little with the program: it does not instrument every call, so the profile resembles the unprofiled program, and some can attach to a running process
*Answer D: It interferes little with the program: it does not instrument every call, so the profile resembles the unprofiled program, and some can attach to a running process.* cProfile measures every call and return, which adds a fixed cost per call and penalizes code with many small calls the most. Sampling looks at the stack every so often: it loses the exact counts, but distorts less and can be used in production.
5. After compiling the loop with Numba, preparing the arrays (rolling maxima, session changes) takes longer than the loop. What makes sense to do now?
- A. Rewrite the loop in Rust
- B. Also compile the loop with Cython
- C. Measure the whole program again: if it is fast enough, stop; if not, profile again, because the bottleneck is now somewhere else
- D. Increase the number of Numba threads
*Answer C: Measure the whole program again: if it is fast enough, stop; if not, profile again, because the bottleneck is now somewhere else.* It is Amdahl's law in practice: when the loop costs tenths of a millisecond, speeding it up further changes nothing. The next step is dictated by a new profile, not by intuition.
## Key points
- First measure the whole program and profile it; then optimize what the profile points to, and measure again. Amdahl's law says the most that can be gained.
- cProfile gives exact call counts (useful for reconstructing what the code does); pyinstrument and py-spy sample the stack with less interference. Python 3.15 ships its own sampling profiler, `profiling.sampling`.
- In the lesson's backtest, more than 80% of the time was spent indexing pandas row by row. Taking pandas out of the loop, with the same results, sped it up by two orders of magnitude.
- Numba compiles the same Python loop over NumPy arrays and Cython compiles it with declared types; both give the same results and run hundreds of times faster than the Python loop.
- Rust with PyO3 and maturin is the option for large, distributable, memory-safe libraries (Polars, Ruff, uv, py-spy); for a research loop, Numba is enough.
- Timings are not shared: they change with the machine. What is shared are the orders of magnitude and the checks that the result does not change.
## Resources
### Documentation
- [Python 3.15: profiling.sampling, statistical profiler (Tachyon)](https://docs.python.org/3.15/library/profiling.sampling.html) · docs.python.org
- [PEP 799: a `profiling` package for Python's profilers](https://peps.python.org/pep-0799/) · peps.python.org
- [Numba: compiling Python code with @jit](https://numba.readthedocs.io/en/stable/user/jit.html) · numba.readthedocs.io
- [Cython: typed memoryviews](https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html) · cython.readthedocs.io
- [py-spy: sampling profiler for Python](https://github.com/benfred/py-spy) · github.com
- [PyO3: user guide](https://pyo3.rs/) · pyo3.rs
### Data
- [Level 5 data: one-minute bars of a future (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/hf5_barras_1min.parquet) · Download · hf5_barras_1min.parquet
Previous lesson: [7.1.1 · Project structure, configuration and quality: from notebook to repository](https://pythonparatrading.com/en/python-trading-course/level-7/python-project-structure-configuration-quality/)
Next lesson: [7.1.3 · Patterns: events, message queues and hexagonal architecture](https://pythonparatrading.com/en/python-trading-course/level-7/event-driven-queues-hexagonal-architecture/)