Lesson 6.5.3 · 70 min

Risks of AI-generated code

The code an assistant writes runs, looks reasonable and is sometimes wrong in ways that raise no errors. This lesson brings together the three risks that cost the most in a trading repository: plausible but incorrect functions, dependencies that do not exist or that someone registered after a model invented them, and secrets that end up in the context, in the code or in the logs. Each one with its published evidence and a defense that runs.

Fig. — Drawdown of the equal-weight portfolio of the level 2 universe computed with the assistant's function (summed returns) and the correct one (compounded returns): the worst drawdown goes from 63.1% to 95.3%
Level 6 lessons
  1. 6.1 · Classical and modern financial NLP

    1. 6.1.1 Tokenization, embeddings, financial dictionaries and FinBERT
    2. 6.1.2 Text sources: EDGAR, CNMV, transcripts and social media
    3. 6.1.3 Event studies with text signals, timestamp alignment and topic modeling
  2. 6.2 · LLMs for research and trading

    1. 6.2.1 How an LLM works: from the next token to the API
    2. 6.2.2 Structured prompting: JSON you can validate
    3. 6.2.3 Guidance, KPIs and tone: what an LLM extracts well
    4. 6.2.4 Lightweight fine-tuning for financial tasks: LoRA and QLoRA
    5. 6.2.5 Look-ahead bias in LLMs: the model that already knows the future
  3. 6.3 · Financial RAG

    1. 6.3.1 Embeddings and vector databases: FAISS, Chroma, pgvector and Qdrant
    2. 6.3.2 Chunking financial documents: tables, footnotes and XBRL
    3. 6.3.3 RAG over reports, prospectuses and KIDs, and how to evaluate it
  4. 6.4 · AI agents for trading

    1. 6.4.1 Tool-calling agents: tool calling, ReAct and planning
    2. 6.4.2 Agent frameworks and MCP
    3. 6.4.3 TradingAgents-style multi-agent architectures
    4. 6.4.4 Evidence and limits of agents
    5. 6.4.5 Guardrails for agents: the agent proposes, the engine decides
    6. 6.4.6 Research agents: hypotheses, alpha mining and review
  5. 6.5 · AI as the quant's copilot

    1. 6.5.1 Coding assistants with a test-first workflow
    2. 6.5.2 Automated review of backtests and code migrations
    3. 6.5.3 Risks of AI-generated code
  6. 6.6 · Big data for trading

    1. 6.6.1 When data becomes “big”: ticks, order books, options and universes
    2. 6.6.2 Parquet, Arrow, Polars streaming and out-of-core DuckDB
    3. 6.6.3 Dask, Spark and Ray: from one machine to a cluster
    4. 6.6.4 Time series databases: TimescaleDB, QuestDB, ClickHouse, ArcticDB and kdb+
    5. 6.6.5 Cloud storage, lakehouse (Delta, Iceberg) and GPU (RAPIDS, cuDF)
  7. 6.7 · Alternative data

    1. 6.7.1 Overview and size of the alternative data market
    2. 6.7.2 Types of alternative data: what each one measures
    3. 6.7.3 Responsible web scraping
    4. 6.7.4 Alternative data vendors and due diligence
    5. 6.7.5 From alternative data to a trading signal
    6. 6.7.6 Satellite imagery, computer vision and supply chains

Three risks the usual tests do not see

Lessons 6.5.1 and 6.5.2 made sure that an assistant's code does what was asked and does not look into the future. Three risks remain that do not depend on the strategy, but on how the code is produced:

AssistantFunctionsDependenciesContext and diffPlausible and wrongInvented packageLeaked keyHand-written testsIndex and lockfileDiff scan
Mermaid source
flowchart TB
  A[Assistant] --> F[Functions]
  A --> D[Dependencies]
  A --> S[Context and diff]
  F --> F1[Plausible and wrong]
  D --> D1[Invented package]
  S --> S1[Leaked key]
  F1 --> G[Hand-written tests]
  D1 --> H[Index and lockfile]
  S1 --> K[Diff scan]

These are not theoretical risks. There are published studies with figures, all with models that have since been superseded, so they serve to size the problem, not to measure today's assistants:

StudyWhat it measuredResult
Pearce et al., IEEE S&P 20221,689 programs generated by GitHub Copilot in 89 security scenariosAround 40% vulnerable
Perry et al., ACM CCS 2023A user study, with and without an assistantWith the assistant they wrote less secure code and more often believed it was secure
Spracklen et al., USENIX Security 2025576,000 code samples from 16 models in two languagesAverage rate of invented packages of at least 5.2% in commercial models and 21.7% in open ones; 205,474 distinct invented names
Lanyado, Lasso Security, 2024Registered an empty package under a name that models inventedMore than 30,000 real downloads in three months

The second row is the one that matters most in practice: the risk is not only that the code is wrong, but that whoever receives it trusts it more.

Plausible but incorrect code

Four functions that an assistant proposes for a metrics library. All of them run, all return reasonable numbers and all have a bug that produces no failure:

IN [01]python
import numpy as np
import pandas as pd

def max_drawdown(r):
    """Maximum drop from a peak, as a fraction."""
    curva = r.cumsum()
    return float((curva - curva.cummax()).min())

def sharpe_anual(r, rf_anual=0.0):
    """Annualized Sharpe of daily returns."""
    exceso = r - rf_anual
    return float(exceso.mean() / exceso.std() * np.sqrt(252))

def vwap_intradia(op):
    """Cumulative VWAP of each session (columns ts, precio,
    volumen; ts in UTC)."""
    pv = (op["precio"] * op["volumen"]).cumsum()
    return pv / op["volumen"].cumsum()

def a_utc(ts):
    """New York times without a zone, to UTC."""
    return ts.dt.tz_localize("UTC")

r = pd.Series([0.10, -0.10, -0.10])
print(f"drawdown {max_drawdown(r):.4f}  ·  "
      f"Sharpe {sharpe_anual(r):.2f}")
OUT [01]The four functions the assistant proposes
drawdown -0.2000  ·  Sharpe -4.58

The bugs, one by one: the drawdown adds simple returns instead of compounding them; the Sharpe subtracts an annual risk-free rate from daily returns; the VWAP accumulates from the start of the DataFrame and does not reset each session; and the conversion to UTC declares that the times are already UTC instead of localizing them in New York and converting them. The drawdown in the cell, −0.2000, looks right. The real drop from 1.10 to 0.891 is 19%.

The tests that catch them are written as in lesson 6.5.1: by hand, from the definition, and at the edges. For each function, a case computed by hand and, when one exists, a property that must always hold:

IN [02]python
def ops(*filas):
    return pd.DataFrame(filas, columns=["ts", "precio", "volumen"]
                        ).assign(ts=lambda d: pd.to_datetime(
                            d["ts"], utc=True))

TESTS = {
    "drawdown: three days by hand": lambda f: np.isclose(
        f["max_drawdown"](pd.Series([0.10, -0.10, -0.10])),
        0.891 / 1.1 - 1),
    "drawdown: negative first day": lambda f: np.isclose(
        f["max_drawdown"](pd.Series([-0.10, 0.05])), -0.10),
    "drawdown: between -1 and 0": lambda f: -1 <= f["max_drawdown"](
        pd.Series([0.5, -0.6, -0.6])) <= 0,
    "Sharpe: zero mean excess": lambda f: abs(f["sharpe_anual"](
        pd.Series([0.03 / 252 + 0.01, 0.03 / 252 - 0.01] * 50),
        rf_anual=0.03)) < 1e-9,
    "VWAP: resets each session": lambda f: np.isclose(
        f["vwap_intradia"](ops(
            ("2024-01-02 15:00", 10.0, 100),
            ("2024-01-02 16:00", 11.0, 100),
            ("2024-01-03 15:00", 20.0, 50))).iloc[-1], 20.0),
    "UTC: winter and summer": lambda f: list(
        f["a_utc"](pd.Series(pd.to_datetime(
            ["2024-01-02 09:30", "2024-03-11 09:30"])))
        .dt.strftime("%H:%M")) == ["14:30", "13:30"],
}

def pasar(funciones):
    return {nombre: bool(t(funciones)) for nombre, t in TESTS.items()}

propuesta = {"max_drawdown": max_drawdown,
             "sharpe_anual": sharpe_anual,
             "vwap_intradia": vwap_intradia, "a_utc": a_utc}
for nombre, ok in pasar(propuesta).items():
    print(f"{'pass ' if ok else 'FAIL '}  {nombre}")
OUT [02]Hand-written tests
FAIL   drawdown: three days by hand
FAIL   drawdown: negative first day
FAIL   drawdown: between -1 and 0
FAIL   Sharpe: zero mean excess
FAIL   VWAP: resets each session
FAIL   UTC: winter and summer

The four functions fail all six tests. None of them is sophisticated: three days of returns computed by hand, a negative first day, the property that a drawdown lies between −1 and 0 (with +50%, −60% and −60%, the assistant's function gives −120%), two returns symmetric around the daily risk-free rate (zero Sharpe), a second session with a single trade (its VWAP is its price) and two dates on either side of the US daylight saving change of March 10, 2024 (9:30 in New York is 14:30 UTC in January and 13:30 in March). What they have in common is that they are written from the definition, without looking at the code.

The first fix of the drawdown that occurs to almost everyone, and to almost every assistant, is to compound:

IN [03]python
def max_drawdown_v2(r):
    curva = (1 + r).cumprod()
    return float((curva / curva.cummax() - 1).min())

v2 = dict(propuesta, max_drawdown=max_drawdown_v2)
{k: v for k, v in pasar(v2).items() if k.startswith("drawdown")}
OUT [03]First fix of the drawdown
{'drawdown: three days by hand': True, 'drawdown: negative first day': False, 'drawdown: between -1 and 0': True}

It passes two of the three drawdown tests and fails the one with the negative first day: the curve starts at 0.9 and so does its running maximum, so the drop from the starting capital is not seen. It is a bug inside the fix of another, and in a backtest that starts by losing it underestimates the worst drawdown. The good version adds the starting point (capital 1) before compounding. The four fixed functions pass all six tests:

IN [04]python
def max_drawdown_ok(r):
    curva = pd.concat([pd.Series([1.0]),
                       (1 + r).cumprod()], ignore_index=True)
    return float((curva / curva.cummax() - 1).min())

def sharpe_anual_ok(r, rf_anual=0.0):
    exceso = r - rf_anual / 252
    return float(exceso.mean() / exceso.std() * np.sqrt(252))

def vwap_intradia_ok(op):
    sesion = (op["ts"].dt.tz_convert("America/New_York")
              .dt.normalize())
    pv = (op["precio"] * op["volumen"]).groupby(sesion).cumsum()
    return pv / op["volumen"].groupby(sesion).cumsum()

def a_utc_ok(ts):
    return (ts.dt.tz_localize("America/New_York")
            .dt.tz_convert("UTC"))

corregidas = {"max_drawdown": max_drawdown_ok,
              "sharpe_anual": sharpe_anual_ok,
              "vwap_intradia": vwap_intradia_ok, "a_utc": a_utc_ok}
print(all(pasar(corregidas).values()))
OUT [04]The four fixed functions
True

How much do these bugs matter with data? The equal-weighted portfolio of the stocks in the level 2 universe (synthetic data, from 2006 to 2025, with the 2008 crisis), with a risk-free rate of 2% a year:

IN [05]python
ajustados = pd.read_parquet("universo2_ajustados.parquet")
P = ajustados.pivot(index="fecha", columns="ticker",
                    values="cierre_ajustado")
cartera = P.pct_change(fill_method=None).mean(axis=1).iloc[1:]

for nombre, mal, bien in [
        ("worst drawdown", max_drawdown(cartera),
         max_drawdown_ok(cartera)),
        ("Sharpe with rf 2%", sharpe_anual(cartera, 0.02),
         sharpe_anual_ok(cartera, 0.02))]:
    print(f"{nombre:<18} assistant {mal:8.3f}   correct {bien:7.3f}")
OUT [05]The bugs on the level 2 portfolio
worst drawdown     assistant   -0.953   correct  -0.631
Sharpe with rf 2%  assistant  -24.659   correct   0.290

With data, the two bugs stop being details (the figure at the top shows the drawdown over the whole series). The portfolio's worst drawdown with the assistant's function is 95.3%, against a real 63.1%: adding simple returns over a long decline exaggerates the loss (and over long rallies it understates it, as the figure shows between 2010 and 2018). And the Sharpe with a risk-free rate of 2% a year goes from 0.290 to −24.659: subtracting 2% from every daily return is subtracting 504% a year. The second bug is so large that someone would see it; the first, in a report with many figures, not necessarily.

There is a fifth kind of bug, the easiest: the invented API. An assistant writes a call with a plausible name that the library does not have. It is the good case, because it fails as soon as it runs:

IN [06]python
cartera.rolling(20).zscore()
OUT [06]An API that does not exist
AttributeError: 'Rolling' object has no attribute 'zscore'

The danger is not the code that fails, but the code that works. That is the reason for the order of lessons 6.5.1 to 6.5.3: tests are written first and by hand, cases are chosen at the edges (first day, daylight saving change, second session) and properties are always checked (a drawdown cannot go beyond −100%).

Invented dependencies

A model that generates code also generates its import statements. If the name it invents sounds right, the person who receives it does the natural thing: pip install with that name. If the package does not exist, the command fails and nothing happens. The risk starts when someone registers that name on the public index with malicious code, knowing that models repeat it. Spracklen et al. describe package hallucinations as a persistent, systematic phenomenon of code-generating models, not as a rare accident. It is a variant of typosquatting (registering names similar to those of popular packages to exploit typos) that needs no typo: the model supplies the name.

There is an older and subtler trap: the name in the import is not always the name of the package. import sklearn comes from scikit-learn; the sklearn package on PyPI exists, but its description says “deprecated sklearn package, use scikit-learn instead” (consulted on October 23, 2026).

The defense is to check every new dependency before installing it. PyPI publishes the metadata of each project in a JSON API (/pypi/<project>/json, with the description, the versions and the upload date of each file), and a project that does not exist returns 404. The cell starts a simulated index with that same response shape, with no network. The six real packages carry their PyPI data from October 23, 2026 (first upload and number of versions); quantmetrics-ext and pandsa are fictitious, registered in the simulator 6 and 20 days ago. On the real PyPI, on that date, neither of them existed.

IN [07]python
import json
import threading
from datetime import date, timedelta
from http.server import BaseHTTPRequestHandler, HTTPServer

HOY = date(2026, 10, 23)        # fixed date: nothing depends on the clock

def proyecto(nombre, resumen, primera, versiones):
    dias = np.linspace(0, (HOY - primera).days, versiones)
    return {"info": {"name": nombre, "summary": resumen},
            "releases": {f"0.{i}": [{"upload_time_iso_8601":
                         f"{primera + timedelta(days=int(d))}"
                         "T12:00:00Z"}] for i, d in enumerate(dias)}}

INDICE = {p["info"]["name"]: p for p in [
    proyecto("pandas", "Powerful data structures for data "
             "analysis", date(2009, 12, 25), 120),
    proyecto("numpy", "Fundamental package for array computing",
             date(2006, 12, 2), 152),
    proyecto("scikit-learn", "A set of python modules for machine "
             "learning", date(2011, 9, 22), 76),
    proyecto("sklearn", "deprecated sklearn package, use "
             "scikit-learn instead", date(2015, 7, 15), 10),
    proyecto("quantstats", "Portfolio analytics for quants",
             date(2019, 5, 1), 82),
    proyecto("pyarrow", "Python library for Apache Arrow",
             date(2017, 3, 16), 66),
    proyecto("quantmetrics-ext", "Quant metrics extensions",
             HOY - timedelta(days=6), 1),
    proyecto("pandsa", "Data analysis toolkit",
             HOY - timedelta(days=20), 1),
]}

class Indice(BaseHTTPRequestHandler):
    def do_GET(self):
        partes = self.path.strip("/").split("/")   # pypi/<p>/json
        datos = INDICE.get(partes[1]) if len(partes) == 3 else None
        cuerpo = json.dumps(datos).encode() if datos else b"{}"
        self.send_response(200 if datos else 404)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(cuerpo)

    def log_message(self, *args):
        pass

servidor = HTTPServer(("127.0.0.1", 0), Indice)
threading.Thread(target=servidor.serve_forever, daemon=True).start()
URL = f"http://127.0.0.1:{servidor.server_port}"
print(len(INDICE), "projects in the simulated index")
OUT [07]A simulated package index
8 projects in the simulated index

The dependency reviewer queries each name and applies five checks: that it exists, that it is not too new, that it has a history (more than one version), that it is not marked as deprecated and that it does not look too much like the name of a popular package without being it:

IN [08]python
import difflib
import urllib.error
import urllib.request

POPULARES = ["pandas", "numpy", "scikit-learn", "scipy",
             "matplotlib", "pyarrow", "polars", "requests"]

def consultar(nombre):
    try:
        with urllib.request.urlopen(f"{URL}/pypi/{nombre}/json",
                                    timeout=5) as r:
            return json.load(r)
    except urllib.error.HTTPError as e:
        if e.code == 404:
            return None
        raise

def revisar(nombre, dias_minimos=90):
    datos = consultar(nombre)
    if datos is None:
        return "DOES NOT EXIST: do not install (invented name?)"
    subidas = [f["upload_time_iso_8601"][:10]
               for fs in datos["releases"].values() for f in fs]
    edad = (HOY - date.fromisoformat(min(subidas))).days
    avisos = []
    if edad < dias_minimos:
        avisos.append(f"published {edad} days ago")
    if len(datos["releases"]) < 2:
        avisos.append("a single version")
    if "deprecated" in datos["info"]["summary"].lower():
        avisos.append("deprecated: " + datos["info"]["summary"])
    parecido = difflib.get_close_matches(nombre, POPULARES, 1, 0.8)
    if parecido and parecido[0] != nombre:
        avisos.append(f"resembles {parecido[0]}")
    return "; ".join(avisos) or f"ok ({edad // 365} years)"

PROPUESTA = ["pandas", "numpy", "sklearn", "quantmetrics-ext",
             "finsignal-toolkit", "pandsa", "pyarrow"]
for nombre in PROPUESTA:
    print(f"{nombre:<18} {revisar(nombre)}")
OUT [08]Reviewing the dependencies the assistant proposes
pandas             ok (16 years)
numpy              ok (19 years)
sklearn            deprecated: deprecated sklearn package, use scikit-learn instead
quantmetrics-ext   published 6 days ago; a single version
finsignal-toolkit  DOES NOT EXIST: do not install (invented name?)
pandsa             published 20 days ago; a single version; resembles pandas
pyarrow            ok (9 years)
Fig. — The dependencies the assistant proposes in the simulated index: age and number of releases. Established packages sit top right; those registered days ago, with one release, bottom left. finsignal-toolkit is missing: it does not exist

The reviewer lets the three established packages through and stops the other four, each for a different reason. sklearn exists and is ten years old, but it is marked as deprecated: the assistant wanted scikit-learn. finsignal-toolkit does not exist: it is the harmless case, as long as nobody registers it. quantmetrics-ext is the attack case: a name a model would invent, registered 6 days ago, with a single version. And pandsa, registered 20 days ago with one version, resembles pandas: it is a typo, whether it comes from a person or from a model.

None of those signals proves that a package is malicious, and a new package can be legitimate. What they do is stop the automatic installation until a person looks at where it comes from: its repository, who publishes it, whether the project exists outside the index.

Lockfiles, hashes and waiting periods

Lesson 0.2.2 introduced uv.lock: the file that pins the exact version of every package in the environment, its source and the sha256 of each file; it also stores the upload date of each file to the index. It is what makes the course reproducible: the uv.lock of the main environment pins more than 160 packages. For security, the lockfile has another virtue: it turns every change of dependencies into a reviewable diff. When an assistant adds a package, the new lockfile says exactly what comes in, from where and from when.

The cell audits a lockfile change with the shape of uv.lock ([[package]] tables with name, version, source and files with hash and upload-time; the hashes are shortened):

IN [09]python
import tomllib

ANTES = """
[[package]]
name = "pandas"
version = "3.0.6"
source = { registry = "https://pypi.org/simple" }
wheels = [
  { hash = "sha256:9f1c…", upload-time = "2026-09-30T10:01:12Z" },
]
"""
DESPUES = ANTES + """
[[package]]
name = "quantmetrics-ext"
version = "0.0.1"
source = { registry = "https://pypi.org/simple" }
wheels = [
  { hash = "sha256:52ab…", upload-time = "2026-10-17T21:40:03Z" },
]

[[package]]
name = "finsignal"
version = "1.2.0"
source = { git = "https://example.com/u/finsignal.git" }
"""

def paquetes(texto):
    return {p["name"]: p for p in tomllib.loads(texto)["package"]}

def auditar(antes, despues, espera_dias=7):
    viejos, nuevos = paquetes(antes), paquetes(despues)
    for nombre in sorted(set(nuevos) - set(viejos)):
        p = nuevos[nombre]
        avisos = []
        if p["source"].get("registry") != "https://pypi.org/simple":
            avisos.append(f"source outside PyPI: {p['source']}")
        ficheros = p.get("wheels", []) + (
            [p["sdist"]] if "sdist" in p else [])
        if not ficheros or not all("hash" in f for f in ficheros):
            avisos.append("no hash")
        for f in ficheros:
            subida = date.fromisoformat(f["upload-time"][:10])
            if (HOY - subida).days < espera_dias:
                avisos.append(f"uploaded {(HOY - subida).days} "
                              "days ago")
        print(f"new {nombre} {p['version']}: "
              f"{'; '.join(avisos) or 'ok'}")

auditar(ANTES, DESPUES)
OUT [09]Auditing a proposal's lockfile
new finsignal 1.2.0: source outside PyPI: {'git': 'https://example.com/u/finsignal.git'}; no hash
new quantmetrics-ext 0.0.1: uploaded 6 days ago

Two new packages and two different warnings. finsignal does not come from PyPI but from a git repository, without a hash: what gets installed will depend on what is in that repository on the day of installation. quantmetrics-ext comes from PyPI with its hash, but its only file was uploaded 6 days ago. The hash guarantees that everyone installs exactly the same thing; it says nothing about whether that thing is legitimate. That is why the two checks go together.

In a repository, those checks translate into a few uv and pip commands and options, all from their documentation:

uv add pandas                    # resolves and writes pyproject.toml and uv.lock
uv sync --locked                 # in CI: fails if uv.lock is not up to date
uv pip compile requirements.in --generate-hashes -o requirements.txt
pip install --require-hashes -r requirements.txt

And in pyproject.toml, a waiting period: uv only considers files uploaded before the given date or age (exclude-newer, which accepts durations such as 7 days), so a version published yesterday does not enter the environment until that period has passed:

IN [10]toml
[tool.uv]
exclude-newer = "7 days"

Secrets in the context, in the code and in the logs

Lesson 4.4.5 showed where to store a broker's credentials, how to redact them in logs and how to scan a repository before the commit. A coding assistant adds new paths through which a key can leak:

Project .envContextProviderGenerated codeRepositoryLogs
Mermaid source
flowchart TB
  E[Project .env] --> C[Context]
  C --> P[Provider]
  C --> G[Generated code]
  G --> R[Repository]
  C --> L[Logs]
  • Context: an assistant reads project files to understand the code. If it reads a .env, the key travels to the model provider inside the context.
  • Generated code: if the key was in the context, the model can write it into the code “so that it works”, into a test or into an example.
  • Logs: agent sessions store what they read and what they ran; an env or a print of the configuration leaves the key in those logs.

The tools have rules to exclude files, with nuances worth reading in their documentation (consulted on October 23, 2026). Claude Code's shows how to prevent it from reading .env files with deny rules in its configuration ("deny": ["Read(./.env)", "Read(./.env.*)"]). Cursor's describes .cursorignore, which blocks files for its AI features, but warns that complete protection is not guaranteed and that the agent's terminal and MCP tools cannot block access to those files. The conclusion is that of lesson 4.4.5: the key must not be in the project tree, but in the system's or the cloud's secret manager, and whatever is in the tree is treated as if it were going to leak.

Two defenses that run. The first filters the context before sending it to a model: it leaves out the files that match secret patterns and redacts in the rest whatever looks like a key. The toy keys start with FALSA_ (Spanish for “fake”), as in lesson 4.4.5, and are not printed: only their fingerprint.

IN [11]python
import fnmatch
import hashlib
import math
import re
from collections import Counter

def falsa(nombre, largo=40):
    return "FALSA_" + hashlib.sha256(nombre.encode()).hexdigest()[
        : largo - 6]

def huella(valor):
    return hashlib.sha256(valor.encode()).hexdigest()[:8]

CLAVE = falsa("BROKER_SECRET")
REPO = {
    ".env": f"BROKER_SECRET={CLAVE}\n",
    "config.py": 'BROKER_URL = "https://api.broker.example"\n'
                 'BROKER_SECRET = os.environ["BROKER_SECRET"]\n',
    "senales.py": "def momentum(p, n=20):\n"
                  "    return p.pct_change(n)\n",
    "notas.md": f"tested with the key {CLAVE[:30]}… and it works\n",
}
EXCLUIR = [".env", ".env.*", "*.pem", "*secret*"]
TOKEN = re.compile(r"[A-Za-z0-9_+/=-]{20,}")

def entropia(t):
    n = len(t)
    return -sum(c / n * math.log2(c / n) for c in Counter(t).values())

def contexto_seguro(repo):
    partes, informe = [], []
    for nombre, texto in sorted(repo.items()):
        if any(fnmatch.fnmatch(nombre, g) for g in EXCLUIR):
            informe.append(f"{nombre}: excluded")
            continue
        def tachar(m):
            t = m.group(0)
            if t.startswith("FALSA_") or entropia(t) >= 3.5:
                informe.append(f"{nombre}: redacted {huella(t)}")
                return "[REDACTED]"
            return t
        partes.append(f"# {nombre}\n{TOKEN.sub(tachar, texto)}")
    return "\n".join(partes), informe

enviar, informe = contexto_seguro(REPO)
print("\n".join(informe))
print("any key in what is sent?", CLAVE[:20] in enviar)
OUT [11]Filtering the context before sending it
.env: excluded
notas.md: redacted 5ce46225
any key in what is sent? False

The filter leaves out the whole .env and redacts the key that someone pasted into notas.md, a common leak: notes, drafts and saved notebook outputs. config.py goes through unchanged because it reads the key from the environment and does not contain it. What is sent carries no key. The report identifies what was redacted by its fingerprint, as in lesson 4.4.5, so that you can search for where else it appears without writing it again.

The second defense reviews what the assistant returns: only the lines added in its proposal, which is what would reach the repository. In the example, to “simplify” the configuration, the assistant has replaced the read of the environment variable with the value it saw in the context of another session:

IN [12]python
import difflib as dl

propuesta_config = ('BROKER_URL = "https://api.broker.example"\n'
                    f'BROKER_SECRET = "{CLAVE}"  # simplified\n')

def secretos_en_diff(antes, despues, nombre):
    hallazgos = []
    for linea in dl.unified_diff(antes.splitlines(),
                                 despues.splitlines(), lineterm=""):
        if not linea.startswith("+") or linea.startswith("+++"):
            continue
        for t in TOKEN.findall(linea):
            if t.startswith("FALSA_") or entropia(t) >= 3.5:
                hallazgos.append(f"{nombre}: +{len(t)} characters, "
                                 f"fingerprint {huella(t)}")
    return hallazgos

print(secretos_en_diff(REPO["config.py"], propuesta_config,
                       "config.py") or "no findings")
print(secretos_en_diff(REPO["senales.py"], REPO["senales.py"]
                       + "def media(p, n=20):\n"
                         "    return p.rolling(n).mean()\n",
                       "senales.py") or "no findings")
OUT [12]Scanning the proposal's diff
['config.py: +40 characters, fingerprint 3f1154ad']
no findings

The diff scanner finds the key in the added line of config.py and raises no warnings on the legitimate change to senales.py. Looking only at the added lines has two advantages: it is fast and it does not repeat the warnings already accepted for the rest of the repository. If the key did get written into the repository, even if it is deleted later, it has to be rotated: it is still in the history (lesson 4.4.5).

A checklist for every proposal

RiskCheckWho does it
Plausible and wrong codeHand-written edge-case tests and properties, written first (lessons 6.5.1 and this one)Person; CI runs them
Invented APIRun it: it fails on its ownCI
Backtest biasesStatic reviewer, truncation and placebo (lesson 6.5.2)CI; the person reads the report
New dependencyExists, age, versions, deprecated, similar name; read its repositoryPerson, with the help of a script
LockfileDiff of uv.lock: source, hashes, upload date; uv sync --lockedCI
Freshly published versionWaiting period (exclude-newer)Configuration
Known vulnerabilitiespip-audit or equivalentCI
Secrets in the contextOutside the tree; context filter; assistant rulesConfiguration and person
Secrets in the diffScan of the added lines before the commit (lesson 4.4.5)Pre-commit hook and CI

None of these checks needs to trust the assistant, and none asks it to review itself. It is the same rule as for the agents in lesson 6.4.5: the model proposes and a deterministic process decides.

Exercises

  1. An assistant proposes vol_anual = r.std() * np.sqrt(365) for stocks that trade about 252 sessions a year. Write a hand-written test that catches it and compute how wrong it is on the level 2 portfolio.
Solution
IN [13]python
def vol_anual_asistente(r):
    return float(r.std() * np.sqrt(365))

def vol_anual(r, sesiones=252):
    return float(r.std() * np.sqrt(sesiones))

# by hand: alternating returns of +1% and -1% -> deviation 0.01
r_test = pd.Series([0.01, -0.01] * 126)
d = r_test.std()
print(f"daily deviation {d:.5f}; expected annual "
      f"{d * np.sqrt(252):.4f}; assistant "
      f"{vol_anual_asistente(r_test):.4f}")
print(f"portfolio: {vol_anual(cartera):.4f} vs "
      f"{vol_anual_asistente(cartera):.4f} "
      f"({vol_anual_asistente(cartera) / vol_anual(cartera) - 1:.1%})")
OUT [13]Annualizing with 365
daily deviation 0.01002; expected annual 0.1591; assistant 0.1914
portfolio: 0.2012 vs 0.2422 (20.4%)

With alternating returns of +1% and −1%, the daily standard deviation is 0.01002 and the correct annual volatility, 0.1591; the assistant's, 0.1914. On the level 2 portfolio, 0.2012 against 0.2422: 20.4% more, which is 365/252−1\sqrt{365/252} - 1. In crypto, which trades every day, 365 would be correct; that is why the number of sessions is a parameter and the test fixes the one for stocks.

  1. Many import names do not match the package name on PyPI (sklearn and scikit-learn, yaml and PyYAML, cv2 and opencv-python). Add to the dependency reviewer a table of equivalences that, before querying the index, translates the import name into the package name and flags the change.
Solution
IN [14]python
IMPORT_A_PAQUETE = {"sklearn": "scikit-learn", "yaml": "PyYAML",
                    "cv2": "opencv-python", "bs4": "beautifulsoup4"}

def revisar_import(nombre_import):
    paquete = IMPORT_A_PAQUETE.get(nombre_import, nombre_import)
    nota = (f" (import {nombre_import} -> package {paquete})"
            if paquete != nombre_import else "")
    return f"{paquete}: {revisar(paquete)}{nota}"

for n in ["sklearn", "pandas", "quantmetrics"]:
    print(revisar_import(n))
OUT [14]From the import to the package
scikit-learn: ok (15 years) (import sklearn -> package scikit-learn)
pandas: ok (16 years)
quantmetrics: DOES NOT EXIST: do not install (invented name?)

sklearn is translated to scikit-learn and passes; pandas does not change; quantmetrics is not in the table and does not exist in the index. The table has to be maintained by hand and is short on purpose: only the known cases. A name that is not in it is queried as is, and if it does not exist, installation stops. What you must never do is “fix” a name that does not exist by searching the index for the closest one and installing it: that is exactly what the attack exploits.

  1. The context filter excludes .env by name. What happens if the key is in config.local.json? Propose two changes: one to the filter and another to the repository.
Solution

The filter excludes by file name, so config.local.json would go through whole. If the key has the shape of the toy ones or a high entropy, redaction by content would catch it anyway; if it is short or readable (a password such as trading2026), it would not. Two changes:

  • In the filter: invert the logic. Instead of a list of exclusions, a list of inclusions (*.py, *.md, *.toml of the source code) and everything else out. What has not been chosen to be sent is not sent.
  • In the repository: the local configuration with secrets should not live in the project tree, but in the secret manager or in a user directory outside the repository (lesson 4.4.5), and config.local.json should be in .gitignore and in the assistant's deny rules as a second line of defense.

Each answer is folded under its question

Self-assessment

  1. An assistant computes the drawdown with curva = r.cumsum() and (curva - curva.cummax()).min(). Why is it dangerous?

    • AIt is not: the sum of returns and their product are equivalent
    • BBecause it is slow on long series
    • CBecause it adds simple returns instead of compounding them: it gives drawdowns in points of cumulative return, not as a percentage of the peak, it can go beyond −100%, and on the course data it exaggerates the portfolio's worst drawdown
    • DBecause cummax does not accept missing values
    Show answer

    Answer C. Because it adds simple returns instead of compounding them: it gives drawdowns in points of cumulative return, not as a percentage of the peak, it can go beyond −100%, and on the course data it exaggerates the portfolio's worst drawdown

    With returns of +50%, −60% and −60%, the real drawdown is 84%; the assistant's function gives −120%, which is impossible. The property test “the drawdown is between −1 and 0” catches it instantly. On the equal-weighted level 2 portfolio, the assistant's version gives a much larger worst drawdown than the real one.

  2. The fixed version of the drawdown uses (1 + r).cumprod() but fails the test with a first day of −10%. Why?

    • ABecause the curve starts at 0.9 and so does the running maximum: the drop from the starting capital of 1 is not seen; the starting point has to be added
    • BBecause cumprod rounds
    • CBecause the test is wrong
    • DBecause negative returns do not compound
    Show answer

    Answer A. Because the curve starts at 0.9 and so does the running maximum: the drop from the starting capital of 1 is not seen; the starting point has to be added

    The curve's previous maximum has to include the starting capital. It is a second bug inside the fix of the first, and only a case written by hand for that edge finds it.

  3. What are “hallucinated” dependencies and why are they a security risk?

    • APackages with programming bugs
    • BPackage names that a model invents when generating code: if someone registers that name on PyPI with malicious code, whoever follows the suggestion installs it. Spracklen et al. (2025) measured an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open models
    • CPackages that the assistant installs without permission
    • DOld versions of real packages
    Show answer

    Answer B. Package names that a model invents when generating code: if someone registers that name on PyPI with malicious code, whoever follows the suggestion installs it. Spracklen et al. (2025) measured an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open models

    The package does not exist until someone registers it. In 2024, a researcher registered an empty package under a name that models invented, huggingface-cli, and it received more than 30,000 real downloads in three months.

  4. An assistant adds quantmetrics-ext to the dependencies. It exists in the index, with a single version published 6 days ago. What do you do?

    • AInstall it: it exists in the index
    • BInstall it with --require-hashes
    • CReplace it with the previous version
    • DNot install it until you know where it comes from: a very recent package, with one version and a name that matches what a model would invent, is the typical shape of the attack; look for the original project and its repository, or write the function
    Show answer

    Answer D. Not install it until you know where it comes from: a very recent package, with one version and a name that matches what a model would invent, is the typical shape of the attack; look for the original project and its repository, or write the function

    Existing in the index says nothing about who published it. Hashes guarantee that you install the same thing that was pinned, not that what was pinned is legitimate. A waiting period (like uv's exclude-newer) also avoids installing what was published a few days ago.

  5. Is a .cursorignore or a read-deny rule enough to keep an assistant from seeing the .env?

    • AYes, they are guarantees
    • BNo: Cursor's documentation says the protection is not guaranteed and that the agent's terminal and MCP tools do not respect the file. The safe approach is for the keys not to be in the working directory and to review the changes before the commit
    • COnly if the repository is private
    • DYes, if the .env is also in .gitignore
    Show answer

    Answer B. No: Cursor's documentation says the protection is not guaranteed and that the agent's terminal and MCP tools do not respect the file. The safe approach is for the keys not to be in the working directory and to review the changes before the commit

    Rules reduce the risk, but an agent with a terminal can read any file the user can read. Secrets go outside the project tree (system or cloud secret manager, lesson 4.4.5) and the diff of every proposal is scanned before the commit.

Key points

  • Plausible but incorrect code does not fail: a drawdown with cumsum, an annual risk-free rate subtracted from daily returns, a VWAP that does not reset each session or a time zone localized as UTC give reasonable, wrong numbers.
  • The tests that catch them are hand-written edge cases (negative first day, daylight saving change, second session) and properties (the drawdown between −1 and 0). The drawdown fix had a second bug that only a hand-written case saw.
  • Invented dependencies are a measured risk: an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open ones (Spracklen et al., USENIX Security 2025). An invented name registered later received more than 30,000 downloads in three months.
  • Every new dependency is checked before installing it: that it exists, its age, its versions, whether it is deprecated and whether it resembles a popular package. Then, a lockfile with exact versions and hashes (uv.lock), --locked in continuous integration and a waiting period for anything just published.
  • Secrets are not protected with assistant rules but by taking them out of the project tree; what is sent as context is filtered and the diff of every proposal is scanned before the commit.

Resources