Lesson 6.5.3 · 70 min
Risks of AI-generated code
The code an assistant writes runs, looks reasonable and is sometimes wrong in ways that raise no errors. This lesson brings together the three risks that cost the most in a trading repository: plausible but incorrect functions, dependencies that do not exist or that someone registered after a model invented them, and secrets that end up in the context, in the code or in the logs. Each one with its published evidence and a defense that runs.
- Module
- 6.5 · AI as the quant's copilot
- Duration
- 70 min
- Prerequisites
- 0.2.2 · Virtual environments and dependencies: uv init, uv add, pyproject.toml and lockfiles, 0.2.6 · Introduction to AI coding assistants: prompts, review and tests, 4.4.5 · Secure credentials: environment variables, secret managers and least privilege, 6.5.1 · Coding assistants with a test-first workflow
- Author
- Jesús Cuesta
- Updated
Level 6 lessons
6.1 · Classical and modern financial NLP
6.2 · LLMs for research and trading
6.3 · Financial RAG
6.4 · AI agents for trading
- 6.4.1 Tool-calling agents: tool calling, ReAct and planning
- 6.4.2 Agent frameworks and MCP
- 6.4.3 TradingAgents-style multi-agent architectures
- 6.4.4 Evidence and limits of agents
- 6.4.5 Guardrails for agents: the agent proposes, the engine decides
- 6.4.6 Research agents: hypotheses, alpha mining and review
6.5 · AI as the quant's copilot
6.6 · Big data for trading
- 6.6.1 When data becomes “big”: ticks, order books, options and universes
- 6.6.2 Parquet, Arrow, Polars streaming and out-of-core DuckDB
- 6.6.3 Dask, Spark and Ray: from one machine to a cluster
- 6.6.4 Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+
- 6.6.5 Cloud storage, lakehouse (Delta, Iceberg) and GPU (RAPIDS, cuDF)
6.7 · Alternative data
Three risks the usual tests do not see
Lessons 6.5.1 and 6.5.2 made sure that an assistant's code does what was asked and does not look into the future. Three risks remain that do not depend on the strategy, but on how the code is produced:
Mermaid source
flowchart TB A[Assistant] --> F[Functions] A --> D[Dependencies] A --> S[Context and diff] F --> F1[Plausible and wrong] D --> D1[Invented package] S --> S1[Leaked key] F1 --> G[Hand-written tests] D1 --> H[Index and lockfile] S1 --> K[Diff scan]
These are not theoretical risks. There are published studies with figures, all with models that have since been superseded, so they serve to size the problem, not to measure today's assistants:
| Study | What it measured | Result |
|---|---|---|
| Pearce et al., IEEE S&P 2022 | 1,689 programs generated by GitHub Copilot in 89 security scenarios | Around 40% vulnerable |
| Perry et al., ACM CCS 2023 | A user study, with and without an assistant | With the assistant they wrote less secure code and more often believed it was secure |
| Spracklen et al., USENIX Security 2025 | 576,000 code samples from 16 models in two languages | Average rate of invented packages of at least 5.2% in commercial models and 21.7% in open ones; 205,474 distinct invented names |
| Lanyado, Lasso Security, 2024 | Registered an empty package under a name that models invented | More than 30,000 real downloads in three months |
The second row is the one that matters most in practice: the risk is not only that the code is wrong, but that whoever receives it trusts it more.
Plausible but incorrect code
Four functions that an assistant proposes for a metrics library. All of them run, all return reasonable numbers and all have a bug that produces no failure:
import numpy as np
import pandas as pd
def max_drawdown(r):
"""Maximum drop from a peak, as a fraction."""
curva = r.cumsum()
return float((curva - curva.cummax()).min())
def sharpe_anual(r, rf_anual=0.0):
"""Annualized Sharpe of daily returns."""
exceso = r - rf_anual
return float(exceso.mean() / exceso.std() * np.sqrt(252))
def vwap_intradia(op):
"""Cumulative VWAP of each session (columns ts, precio,
volumen; ts in UTC)."""
pv = (op["precio"] * op["volumen"]).cumsum()
return pv / op["volumen"].cumsum()
def a_utc(ts):
"""New York times without a zone, to UTC."""
return ts.dt.tz_localize("UTC")
r = pd.Series([0.10, -0.10, -0.10])
print(f"drawdown {max_drawdown(r):.4f} · "
f"Sharpe {sharpe_anual(r):.2f}")drawdown -0.2000 · Sharpe -4.58The bugs, one by one: the drawdown adds simple returns instead of compounding them; the Sharpe subtracts an annual risk-free rate from daily returns; the VWAP accumulates from the start of the DataFrame and does not reset each session; and the conversion to UTC declares that the times are already UTC instead of localizing them in New York and converting them. The drawdown in the cell, −0.2000, looks right. The real drop from 1.10 to 0.891 is 19%.
The tests that catch them are written as in lesson 6.5.1: by hand, from the definition, and at the edges. For each function, a case computed by hand and, when one exists, a property that must always hold:
def ops(*filas):
return pd.DataFrame(filas, columns=["ts", "precio", "volumen"]
).assign(ts=lambda d: pd.to_datetime(
d["ts"], utc=True))
TESTS = {
"drawdown: three days by hand": lambda f: np.isclose(
f["max_drawdown"](pd.Series([0.10, -0.10, -0.10])),
0.891 / 1.1 - 1),
"drawdown: negative first day": lambda f: np.isclose(
f["max_drawdown"](pd.Series([-0.10, 0.05])), -0.10),
"drawdown: between -1 and 0": lambda f: -1 <= f["max_drawdown"](
pd.Series([0.5, -0.6, -0.6])) <= 0,
"Sharpe: zero mean excess": lambda f: abs(f["sharpe_anual"](
pd.Series([0.03 / 252 + 0.01, 0.03 / 252 - 0.01] * 50),
rf_anual=0.03)) < 1e-9,
"VWAP: resets each session": lambda f: np.isclose(
f["vwap_intradia"](ops(
("2024-01-02 15:00", 10.0, 100),
("2024-01-02 16:00", 11.0, 100),
("2024-01-03 15:00", 20.0, 50))).iloc[-1], 20.0),
"UTC: winter and summer": lambda f: list(
f["a_utc"](pd.Series(pd.to_datetime(
["2024-01-02 09:30", "2024-03-11 09:30"])))
.dt.strftime("%H:%M")) == ["14:30", "13:30"],
}
def pasar(funciones):
return {nombre: bool(t(funciones)) for nombre, t in TESTS.items()}
propuesta = {"max_drawdown": max_drawdown,
"sharpe_anual": sharpe_anual,
"vwap_intradia": vwap_intradia, "a_utc": a_utc}
for nombre, ok in pasar(propuesta).items():
print(f"{'pass ' if ok else 'FAIL '} {nombre}")FAIL drawdown: three days by hand
FAIL drawdown: negative first day
FAIL drawdown: between -1 and 0
FAIL Sharpe: zero mean excess
FAIL VWAP: resets each session
FAIL UTC: winter and summerThe four functions fail all six tests. None of them is sophisticated: three days of returns computed by hand, a negative first day, the property that a drawdown lies between −1 and 0 (with +50%, −60% and −60%, the assistant's function gives −120%), two returns symmetric around the daily risk-free rate (zero Sharpe), a second session with a single trade (its VWAP is its price) and two dates on either side of the US daylight saving change of March 10, 2024 (9:30 in New York is 14:30 UTC in January and 13:30 in March). What they have in common is that they are written from the definition, without looking at the code.
The first fix of the drawdown that occurs to almost everyone, and to almost every assistant, is to compound:
def max_drawdown_v2(r):
curva = (1 + r).cumprod()
return float((curva / curva.cummax() - 1).min())
v2 = dict(propuesta, max_drawdown=max_drawdown_v2)
{k: v for k, v in pasar(v2).items() if k.startswith("drawdown")}{'drawdown: three days by hand': True, 'drawdown: negative first day': False, 'drawdown: between -1 and 0': True}It passes two of the three drawdown tests and fails the one with the negative first day: the curve starts at 0.9 and so does its running maximum, so the drop from the starting capital is not seen. It is a bug inside the fix of another, and in a backtest that starts by losing it underestimates the worst drawdown. The good version adds the starting point (capital 1) before compounding. The four fixed functions pass all six tests:
def max_drawdown_ok(r):
curva = pd.concat([pd.Series([1.0]),
(1 + r).cumprod()], ignore_index=True)
return float((curva / curva.cummax() - 1).min())
def sharpe_anual_ok(r, rf_anual=0.0):
exceso = r - rf_anual / 252
return float(exceso.mean() / exceso.std() * np.sqrt(252))
def vwap_intradia_ok(op):
sesion = (op["ts"].dt.tz_convert("America/New_York")
.dt.normalize())
pv = (op["precio"] * op["volumen"]).groupby(sesion).cumsum()
return pv / op["volumen"].groupby(sesion).cumsum()
def a_utc_ok(ts):
return (ts.dt.tz_localize("America/New_York")
.dt.tz_convert("UTC"))
corregidas = {"max_drawdown": max_drawdown_ok,
"sharpe_anual": sharpe_anual_ok,
"vwap_intradia": vwap_intradia_ok, "a_utc": a_utc_ok}
print(all(pasar(corregidas).values()))TrueHow much do these bugs matter with data? The equal-weighted portfolio of the stocks in the level 2 universe (synthetic data, from 2006 to 2025, with the 2008 crisis), with a risk-free rate of 2% a year:
ajustados = pd.read_parquet("universo2_ajustados.parquet")
P = ajustados.pivot(index="fecha", columns="ticker",
values="cierre_ajustado")
cartera = P.pct_change(fill_method=None).mean(axis=1).iloc[1:]
for nombre, mal, bien in [
("worst drawdown", max_drawdown(cartera),
max_drawdown_ok(cartera)),
("Sharpe with rf 2%", sharpe_anual(cartera, 0.02),
sharpe_anual_ok(cartera, 0.02))]:
print(f"{nombre:<18} assistant {mal:8.3f} correct {bien:7.3f}")worst drawdown assistant -0.953 correct -0.631
Sharpe with rf 2% assistant -24.659 correct 0.290With data, the two bugs stop being details (the figure at the top shows the drawdown over the whole series). The portfolio's worst drawdown with the assistant's function is 95.3%, against a real 63.1%: adding simple returns over a long decline exaggerates the loss (and over long rallies it understates it, as the figure shows between 2010 and 2018). And the Sharpe with a risk-free rate of 2% a year goes from 0.290 to −24.659: subtracting 2% from every daily return is subtracting 504% a year. The second bug is so large that someone would see it; the first, in a report with many figures, not necessarily.
There is a fifth kind of bug, the easiest: the invented API. An assistant writes a call with a plausible name that the library does not have. It is the good case, because it fails as soon as it runs:
cartera.rolling(20).zscore()AttributeError: 'Rolling' object has no attribute 'zscore'The danger is not the code that fails, but the code that works. That is the reason for the order of lessons 6.5.1 to 6.5.3: tests are written first and by hand, cases are chosen at the edges (first day, daylight saving change, second session) and properties are always checked (a drawdown cannot go beyond −100%).
Invented dependencies
A model that generates code also generates its import statements. If the name it invents sounds right, the person
who receives it does the natural thing: pip install with that name. If the package does not exist, the command fails
and nothing happens. The risk starts when someone registers that name on the public index with malicious code, knowing
that models repeat it. Spracklen et al. describe package hallucinations as a persistent, systematic phenomenon of
code-generating models, not as a rare accident. It is a variant of typosquatting (registering names similar to those
of popular packages to exploit typos) that needs no typo: the model supplies the name.
There is an older and subtler trap: the name in the import is not always the name of the package. import sklearn
comes from scikit-learn; the sklearn package on PyPI exists, but its description says “deprecated sklearn package,
use scikit-learn instead” (consulted on October 23, 2026).
The defense is to check every new dependency before installing it. PyPI publishes the metadata of each project in
a JSON API (/pypi/<project>/json, with the description, the versions and the upload date of each file), and a
project that does not exist returns 404. The cell starts a simulated index with that same response shape, with no
network. The six real packages carry their PyPI data from October 23, 2026 (first upload and number of versions);
quantmetrics-ext and pandsa are fictitious, registered in the simulator 6 and 20 days ago. On the real PyPI, on
that date, neither of them existed.
import json
import threading
from datetime import date, timedelta
from http.server import BaseHTTPRequestHandler, HTTPServer
HOY = date(2026, 10, 23) # fixed date: nothing depends on the clock
def proyecto(nombre, resumen, primera, versiones):
dias = np.linspace(0, (HOY - primera).days, versiones)
return {"info": {"name": nombre, "summary": resumen},
"releases": {f"0.{i}": [{"upload_time_iso_8601":
f"{primera + timedelta(days=int(d))}"
"T12:00:00Z"}] for i, d in enumerate(dias)}}
INDICE = {p["info"]["name"]: p for p in [
proyecto("pandas", "Powerful data structures for data "
"analysis", date(2009, 12, 25), 120),
proyecto("numpy", "Fundamental package for array computing",
date(2006, 12, 2), 152),
proyecto("scikit-learn", "A set of python modules for machine "
"learning", date(2011, 9, 22), 76),
proyecto("sklearn", "deprecated sklearn package, use "
"scikit-learn instead", date(2015, 7, 15), 10),
proyecto("quantstats", "Portfolio analytics for quants",
date(2019, 5, 1), 82),
proyecto("pyarrow", "Python library for Apache Arrow",
date(2017, 3, 16), 66),
proyecto("quantmetrics-ext", "Quant metrics extensions",
HOY - timedelta(days=6), 1),
proyecto("pandsa", "Data analysis toolkit",
HOY - timedelta(days=20), 1),
]}
class Indice(BaseHTTPRequestHandler):
def do_GET(self):
partes = self.path.strip("/").split("/") # pypi/<p>/json
datos = INDICE.get(partes[1]) if len(partes) == 3 else None
cuerpo = json.dumps(datos).encode() if datos else b"{}"
self.send_response(200 if datos else 404)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(cuerpo)
def log_message(self, *args):
pass
servidor = HTTPServer(("127.0.0.1", 0), Indice)
threading.Thread(target=servidor.serve_forever, daemon=True).start()
URL = f"http://127.0.0.1:{servidor.server_port}"
print(len(INDICE), "projects in the simulated index")8 projects in the simulated indexThe dependency reviewer queries each name and applies five checks: that it exists, that it is not too new, that it has a history (more than one version), that it is not marked as deprecated and that it does not look too much like the name of a popular package without being it:
import difflib
import urllib.error
import urllib.request
POPULARES = ["pandas", "numpy", "scikit-learn", "scipy",
"matplotlib", "pyarrow", "polars", "requests"]
def consultar(nombre):
try:
with urllib.request.urlopen(f"{URL}/pypi/{nombre}/json",
timeout=5) as r:
return json.load(r)
except urllib.error.HTTPError as e:
if e.code == 404:
return None
raise
def revisar(nombre, dias_minimos=90):
datos = consultar(nombre)
if datos is None:
return "DOES NOT EXIST: do not install (invented name?)"
subidas = [f["upload_time_iso_8601"][:10]
for fs in datos["releases"].values() for f in fs]
edad = (HOY - date.fromisoformat(min(subidas))).days
avisos = []
if edad < dias_minimos:
avisos.append(f"published {edad} days ago")
if len(datos["releases"]) < 2:
avisos.append("a single version")
if "deprecated" in datos["info"]["summary"].lower():
avisos.append("deprecated: " + datos["info"]["summary"])
parecido = difflib.get_close_matches(nombre, POPULARES, 1, 0.8)
if parecido and parecido[0] != nombre:
avisos.append(f"resembles {parecido[0]}")
return "; ".join(avisos) or f"ok ({edad // 365} years)"
PROPUESTA = ["pandas", "numpy", "sklearn", "quantmetrics-ext",
"finsignal-toolkit", "pandsa", "pyarrow"]
for nombre in PROPUESTA:
print(f"{nombre:<18} {revisar(nombre)}")pandas ok (16 years)
numpy ok (19 years)
sklearn deprecated: deprecated sklearn package, use scikit-learn instead
quantmetrics-ext published 6 days ago; a single version
finsignal-toolkit DOES NOT EXIST: do not install (invented name?)
pandsa published 20 days ago; a single version; resembles pandas
pyarrow ok (9 years)The reviewer lets the three established packages through and stops the other four, each for a different reason.
sklearn exists and is ten years old, but it is marked as deprecated: the assistant wanted scikit-learn.
finsignal-toolkit does not exist: it is the harmless case, as long as nobody registers it. quantmetrics-ext is the
attack case: a name a model would invent, registered 6 days ago, with a single version. And pandsa, registered 20
days ago with one version, resembles pandas: it is a typo, whether it comes from a person or from a model.
None of those signals proves that a package is malicious, and a new package can be legitimate. What they do is stop the automatic installation until a person looks at where it comes from: its repository, who publishes it, whether the project exists outside the index.
Lockfiles, hashes and waiting periods
Lesson 0.2.2 introduced uv.lock: the file that pins the exact version of every package in the environment, its
source and the sha256 of each file; it also stores the upload date of each file to the index. It is what makes the
course reproducible: the uv.lock of the main environment pins more than 160 packages. For security, the lockfile has
another virtue: it turns every change of dependencies into a reviewable diff. When an assistant adds a package, the
new lockfile says exactly what comes in, from where and from when.
The cell audits a lockfile change with the shape of uv.lock ([[package]] tables with name, version, source
and files with hash and upload-time; the hashes are shortened):
import tomllib
ANTES = """
[[package]]
name = "pandas"
version = "3.0.6"
source = { registry = "https://pypi.org/simple" }
wheels = [
{ hash = "sha256:9f1c…", upload-time = "2026-09-30T10:01:12Z" },
]
"""
DESPUES = ANTES + """
[[package]]
name = "quantmetrics-ext"
version = "0.0.1"
source = { registry = "https://pypi.org/simple" }
wheels = [
{ hash = "sha256:52ab…", upload-time = "2026-10-17T21:40:03Z" },
]
[[package]]
name = "finsignal"
version = "1.2.0"
source = { git = "https://example.com/u/finsignal.git" }
"""
def paquetes(texto):
return {p["name"]: p for p in tomllib.loads(texto)["package"]}
def auditar(antes, despues, espera_dias=7):
viejos, nuevos = paquetes(antes), paquetes(despues)
for nombre in sorted(set(nuevos) - set(viejos)):
p = nuevos[nombre]
avisos = []
if p["source"].get("registry") != "https://pypi.org/simple":
avisos.append(f"source outside PyPI: {p['source']}")
ficheros = p.get("wheels", []) + (
[p["sdist"]] if "sdist" in p else [])
if not ficheros or not all("hash" in f for f in ficheros):
avisos.append("no hash")
for f in ficheros:
subida = date.fromisoformat(f["upload-time"][:10])
if (HOY - subida).days < espera_dias:
avisos.append(f"uploaded {(HOY - subida).days} "
"days ago")
print(f"new {nombre} {p['version']}: "
f"{'; '.join(avisos) or 'ok'}")
auditar(ANTES, DESPUES)new finsignal 1.2.0: source outside PyPI: {'git': 'https://example.com/u/finsignal.git'}; no hash
new quantmetrics-ext 0.0.1: uploaded 6 days agoTwo new packages and two different warnings. finsignal does not come from PyPI but from a git repository, without a
hash: what gets installed will depend on what is in that repository on the day of installation. quantmetrics-ext
comes from PyPI with its hash, but its only file was uploaded 6 days ago. The hash guarantees that everyone installs
exactly the same thing; it says nothing about whether that thing is legitimate. That is why the two checks go
together.
In a repository, those checks translate into a few uv and pip commands and options, all from their documentation:
uv add pandas # resolves and writes pyproject.toml and uv.lock
uv sync --locked # in CI: fails if uv.lock is not up to date
uv pip compile requirements.in --generate-hashes -o requirements.txt
pip install --require-hashes -r requirements.txtAnd in pyproject.toml, a waiting period: uv only considers files uploaded before the given date or age
(exclude-newer, which accepts durations such as 7 days), so a version published yesterday does not enter the
environment until that period has passed:
[tool.uv]
exclude-newer = "7 days"Secrets in the context, in the code and in the logs
Lesson 4.4.5 showed where to store a broker's credentials, how to redact them in logs and how to scan a repository before the commit. A coding assistant adds new paths through which a key can leak:
Mermaid source
flowchart TB E[Project .env] --> C[Context] C --> P[Provider] C --> G[Generated code] G --> R[Repository] C --> L[Logs]
- Context: an assistant reads project files to understand the code. If it reads a
.env, the key travels to the model provider inside the context. - Generated code: if the key was in the context, the model can write it into the code “so that it works”, into a test or into an example.
- Logs: agent sessions store what they read and what they ran; an
envor aprintof the configuration leaves the key in those logs.
The tools have rules to exclude files, with nuances worth reading in their documentation (consulted on October 23,
2026). Claude Code's shows how to prevent it from reading .env files with deny rules in its configuration
("deny": ["Read(./.env)", "Read(./.env.*)"]). Cursor's describes .cursorignore, which blocks files for its AI
features, but warns that complete protection is not guaranteed and that the agent's terminal and MCP tools cannot block
access to those files. The conclusion is that of lesson 4.4.5: the key must not be in the project tree, but in the
system's or the cloud's secret manager, and whatever is in the tree is treated as if it were going to leak.
Two defenses that run. The first filters the context before sending it to a model: it leaves out the files that match
secret patterns and redacts in the rest whatever looks like a key. The toy keys start with FALSA_ (Spanish for
“fake”), as in lesson 4.4.5, and are not printed: only their fingerprint.
import fnmatch
import hashlib
import math
import re
from collections import Counter
def falsa(nombre, largo=40):
return "FALSA_" + hashlib.sha256(nombre.encode()).hexdigest()[
: largo - 6]
def huella(valor):
return hashlib.sha256(valor.encode()).hexdigest()[:8]
CLAVE = falsa("BROKER_SECRET")
REPO = {
".env": f"BROKER_SECRET={CLAVE}\n",
"config.py": 'BROKER_URL = "https://api.broker.example"\n'
'BROKER_SECRET = os.environ["BROKER_SECRET"]\n',
"senales.py": "def momentum(p, n=20):\n"
" return p.pct_change(n)\n",
"notas.md": f"tested with the key {CLAVE[:30]}… and it works\n",
}
EXCLUIR = [".env", ".env.*", "*.pem", "*secret*"]
TOKEN = re.compile(r"[A-Za-z0-9_+/=-]{20,}")
def entropia(t):
n = len(t)
return -sum(c / n * math.log2(c / n) for c in Counter(t).values())
def contexto_seguro(repo):
partes, informe = [], []
for nombre, texto in sorted(repo.items()):
if any(fnmatch.fnmatch(nombre, g) for g in EXCLUIR):
informe.append(f"{nombre}: excluded")
continue
def tachar(m):
t = m.group(0)
if t.startswith("FALSA_") or entropia(t) >= 3.5:
informe.append(f"{nombre}: redacted {huella(t)}")
return "[REDACTED]"
return t
partes.append(f"# {nombre}\n{TOKEN.sub(tachar, texto)}")
return "\n".join(partes), informe
enviar, informe = contexto_seguro(REPO)
print("\n".join(informe))
print("any key in what is sent?", CLAVE[:20] in enviar).env: excluded
notas.md: redacted 5ce46225
any key in what is sent? FalseThe filter leaves out the whole .env and redacts the key that someone pasted into notas.md, a common leak: notes,
drafts and saved notebook outputs. config.py goes through unchanged because it reads the key from the environment and
does not contain it. What is sent carries no key. The report identifies what was redacted by its fingerprint, as in
lesson 4.4.5, so that you can search for where else it appears without writing it again.
The second defense reviews what the assistant returns: only the lines added in its proposal, which is what would reach the repository. In the example, to “simplify” the configuration, the assistant has replaced the read of the environment variable with the value it saw in the context of another session:
import difflib as dl
propuesta_config = ('BROKER_URL = "https://api.broker.example"\n'
f'BROKER_SECRET = "{CLAVE}" # simplified\n')
def secretos_en_diff(antes, despues, nombre):
hallazgos = []
for linea in dl.unified_diff(antes.splitlines(),
despues.splitlines(), lineterm=""):
if not linea.startswith("+") or linea.startswith("+++"):
continue
for t in TOKEN.findall(linea):
if t.startswith("FALSA_") or entropia(t) >= 3.5:
hallazgos.append(f"{nombre}: +{len(t)} characters, "
f"fingerprint {huella(t)}")
return hallazgos
print(secretos_en_diff(REPO["config.py"], propuesta_config,
"config.py") or "no findings")
print(secretos_en_diff(REPO["senales.py"], REPO["senales.py"]
+ "def media(p, n=20):\n"
" return p.rolling(n).mean()\n",
"senales.py") or "no findings")['config.py: +40 characters, fingerprint 3f1154ad']
no findingsThe diff scanner finds the key in the added line of config.py and raises no warnings on the legitimate change to
senales.py. Looking only at the added lines has two advantages: it is fast and it does not repeat the warnings already
accepted for the rest of the repository. If the key did get written into the repository, even if it is deleted later,
it has to be rotated: it is still in the history (lesson 4.4.5).
A checklist for every proposal
| Risk | Check | Who does it |
|---|---|---|
| Plausible and wrong code | Hand-written edge-case tests and properties, written first (lessons 6.5.1 and this one) | Person; CI runs them |
| Invented API | Run it: it fails on its own | CI |
| Backtest biases | Static reviewer, truncation and placebo (lesson 6.5.2) | CI; the person reads the report |
| New dependency | Exists, age, versions, deprecated, similar name; read its repository | Person, with the help of a script |
| Lockfile | Diff of uv.lock: source, hashes, upload date; uv sync --locked | CI |
| Freshly published version | Waiting period (exclude-newer) | Configuration |
| Known vulnerabilities | pip-audit or equivalent | CI |
| Secrets in the context | Outside the tree; context filter; assistant rules | Configuration and person |
| Secrets in the diff | Scan of the added lines before the commit (lesson 4.4.5) | Pre-commit hook and CI |
None of these checks needs to trust the assistant, and none asks it to review itself. It is the same rule as for the agents in lesson 6.4.5: the model proposes and a deterministic process decides.
Exercises
- An assistant proposes
vol_anual = r.std() * np.sqrt(365)for stocks that trade about 252 sessions a year. Write a hand-written test that catches it and compute how wrong it is on the level 2 portfolio.
Solution
def vol_anual_asistente(r):
return float(r.std() * np.sqrt(365))
def vol_anual(r, sesiones=252):
return float(r.std() * np.sqrt(sesiones))
# by hand: alternating returns of +1% and -1% -> deviation 0.01
r_test = pd.Series([0.01, -0.01] * 126)
d = r_test.std()
print(f"daily deviation {d:.5f}; expected annual "
f"{d * np.sqrt(252):.4f}; assistant "
f"{vol_anual_asistente(r_test):.4f}")
print(f"portfolio: {vol_anual(cartera):.4f} vs "
f"{vol_anual_asistente(cartera):.4f} "
f"({vol_anual_asistente(cartera) / vol_anual(cartera) - 1:.1%})")daily deviation 0.01002; expected annual 0.1591; assistant 0.1914
portfolio: 0.2012 vs 0.2422 (20.4%)With alternating returns of +1% and −1%, the daily standard deviation is 0.01002 and the correct annual volatility, 0.1591; the assistant's, 0.1914. On the level 2 portfolio, 0.2012 against 0.2422: 20.4% more, which is . In crypto, which trades every day, 365 would be correct; that is why the number of sessions is a parameter and the test fixes the one for stocks.
- Many
importnames do not match the package name on PyPI (sklearnandscikit-learn,yamlandPyYAML,cv2andopencv-python). Add to the dependency reviewer a table of equivalences that, before querying the index, translates theimportname into the package name and flags the change.
Solution
IMPORT_A_PAQUETE = {"sklearn": "scikit-learn", "yaml": "PyYAML",
"cv2": "opencv-python", "bs4": "beautifulsoup4"}
def revisar_import(nombre_import):
paquete = IMPORT_A_PAQUETE.get(nombre_import, nombre_import)
nota = (f" (import {nombre_import} -> package {paquete})"
if paquete != nombre_import else "")
return f"{paquete}: {revisar(paquete)}{nota}"
for n in ["sklearn", "pandas", "quantmetrics"]:
print(revisar_import(n))scikit-learn: ok (15 years) (import sklearn -> package scikit-learn)
pandas: ok (16 years)
quantmetrics: DOES NOT EXIST: do not install (invented name?)sklearn is translated to scikit-learn and passes; pandas does not change; quantmetrics is not in the table and
does not exist in the index. The table has to be maintained by hand and is short on purpose: only the known cases. A
name that is not in it is queried as is, and if it does not exist, installation stops. What you must never do is “fix”
a name that does not exist by searching the index for the closest one and installing it: that is exactly what the
attack exploits.
- The context filter excludes
.envby name. What happens if the key is inconfig.local.json? Propose two changes: one to the filter and another to the repository.
Solution
The filter excludes by file name, so config.local.json would go through whole. If the key has the shape of the toy
ones or a high entropy, redaction by content would catch it anyway; if it is short or readable (a password such as
trading2026), it would not. Two changes:
- In the filter: invert the logic. Instead of a list of exclusions, a list of inclusions (
*.py,*.md,*.tomlof the source code) and everything else out. What has not been chosen to be sent is not sent. - In the repository: the local configuration with secrets should not live in the project tree, but in the secret
manager or in a user directory outside the repository (lesson 4.4.5), and
config.local.jsonshould be in.gitignoreand in the assistant's deny rules as a second line of defense.
Each answer is folded under its question
Self-assessment
An assistant computes the drawdown with
curva = r.cumsum()and(curva - curva.cummax()).min(). Why is it dangerous?- AIt is not: the sum of returns and their product are equivalent
- BBecause it is slow on long series
- CBecause it adds simple returns instead of compounding them: it gives drawdowns in points of cumulative return, not as a percentage of the peak, it can go beyond −100%, and on the course data it exaggerates the portfolio's worst drawdown
- DBecause
cummaxdoes not accept missing values
Show answer
Answer C. Because it adds simple returns instead of compounding them: it gives drawdowns in points of cumulative return, not as a percentage of the peak, it can go beyond −100%, and on the course data it exaggerates the portfolio's worst drawdown
With returns of +50%, −60% and −60%, the real drawdown is 84%; the assistant's function gives −120%, which is impossible. The property test “the drawdown is between −1 and 0” catches it instantly. On the equal-weighted level 2 portfolio, the assistant's version gives a much larger worst drawdown than the real one.
The fixed version of the drawdown uses
(1 + r).cumprod()but fails the test with a first day of −10%. Why?- ABecause the curve starts at 0.9 and so does the running maximum: the drop from the starting capital of 1 is not seen; the starting point has to be added
- BBecause
cumprodrounds - CBecause the test is wrong
- DBecause negative returns do not compound
Show answer
Answer A. Because the curve starts at 0.9 and so does the running maximum: the drop from the starting capital of 1 is not seen; the starting point has to be added
The curve's previous maximum has to include the starting capital. It is a second bug inside the fix of the first, and only a case written by hand for that edge finds it.
What are “hallucinated” dependencies and why are they a security risk?
- APackages with programming bugs
- BPackage names that a model invents when generating code: if someone registers that name on PyPI with malicious code, whoever follows the suggestion installs it. Spracklen et al. (2025) measured an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open models
- CPackages that the assistant installs without permission
- DOld versions of real packages
Show answer
Answer B. Package names that a model invents when generating code: if someone registers that name on PyPI with malicious code, whoever follows the suggestion installs it. Spracklen et al. (2025) measured an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open models
The package does not exist until someone registers it. In 2024, a researcher registered an empty package under a name that models invented,
huggingface-cli, and it received more than 30,000 real downloads in three months.An assistant adds
quantmetrics-extto the dependencies. It exists in the index, with a single version published 6 days ago. What do you do?- AInstall it: it exists in the index
- BInstall it with
--require-hashes - CReplace it with the previous version
- DNot install it until you know where it comes from: a very recent package, with one version and a name that matches what a model would invent, is the typical shape of the attack; look for the original project and its repository, or write the function
Show answer
Answer D. Not install it until you know where it comes from: a very recent package, with one version and a name that matches what a model would invent, is the typical shape of the attack; look for the original project and its repository, or write the function
Existing in the index says nothing about who published it. Hashes guarantee that you install the same thing that was pinned, not that what was pinned is legitimate. A waiting period (like uv's
exclude-newer) also avoids installing what was published a few days ago.Is a
.cursorignoreor a read-deny rule enough to keep an assistant from seeing the.env?- AYes, they are guarantees
- BNo: Cursor's documentation says the protection is not guaranteed and that the agent's terminal and MCP tools do not respect the file. The safe approach is for the keys not to be in the working directory and to review the changes before the commit
- COnly if the repository is private
- DYes, if the
.envis also in.gitignore
Show answer
Answer B. No: Cursor's documentation says the protection is not guaranteed and that the agent's terminal and MCP tools do not respect the file. The safe approach is for the keys not to be in the working directory and to review the changes before the commit
Rules reduce the risk, but an agent with a terminal can read any file the user can read. Secrets go outside the project tree (system or cloud secret manager, lesson 4.4.5) and the diff of every proposal is scanned before the commit.
Key points
- Plausible but incorrect code does not fail: a drawdown with
cumsum, an annual risk-free rate subtracted from daily returns, a VWAP that does not reset each session or a time zone localized as UTC give reasonable, wrong numbers. - The tests that catch them are hand-written edge cases (negative first day, daylight saving change, second session) and properties (the drawdown between −1 and 0). The drawdown fix had a second bug that only a hand-written case saw.
- Invented dependencies are a measured risk: an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open ones (Spracklen et al., USENIX Security 2025). An invented name registered later received more than 30,000 downloads in three months.
- Every new dependency is checked before installing it: that it exists, its age, its versions, whether it is deprecated and whether it resembles a popular package. Then, a lockfile with exact versions and hashes (
uv.lock),--lockedin continuous integration and a waiting period for anything just published. - Secrets are not protected with assistant rules but by taking them out of the project tree; what is sent as context is filtered and the diff of every proposal is scanned before the commit.
Resources
Documentation
Books
- Spracklen, J. et al. (2025): We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. USENIX Security 2025. arXiv:2406.10279
- Pearce, H. et al. (2022): Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. IEEE S&P 2022. arXiv:2108.09293
- Perry, N. et al. (2023): Do Users Write More Insecure Code with AI Assistants? ACM CCS 2023. arXiv:2211.03622