--- title: "6.5.3 Risks of AI-generated code" description: "Plausible but wrong code caught with tests, invented dependencies checked against a simulated index, lockfiles with hashes and secrets kept out of context." url: "https://pythonparatrading.com/en/python-trading-course/level-6/risks-of-ai-generated-code/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-6/risks-of-ai-generated-code.png" --- # 6.5.3 · Risks of AI-generated code > The code an assistant writes runs, looks reasonable and is sometimes wrong in ways that raise no errors. This lesson brings together the three risks that cost the most in a trading repository: plausible but incorrect functions, dependencies that do not exist or that someone registered after a model invented them, and secrets that end up in the context, in the code or in the logs. Each one with its published evidence and a defense that runs. - Course: [Level 6: Generative AI, LLMs, agents, big data and alternative data](https://pythonparatrading.com/en/python-trading-course/level-6/) - Module: 6.5 · AI as the quant's copilot - Duration: 70 min - Prerequisites: [0.2.2 · Virtual environments and dependencies: uv init, uv add, pyproject.toml and lockfiles](https://pythonparatrading.com/en/python-trading-course/level-0/virtual-environments-dependencies-with-uv/), [0.2.6 · Introduction to AI coding assistants: prompts, review and tests](https://pythonparatrading.com/en/python-trading-course/level-0/ai-coding-assistants/), [4.4.5 · Secure credentials: environment variables, secret managers and least privilege](https://pythonparatrading.com/en/python-trading-course/level-4/secure-api-credentials-and-least-privilege/), [6.5.1 · Coding assistants with a test-first workflow](https://pythonparatrading.com/en/python-trading-course/level-6/test-first-ai-coding-assistants/) ## By the end of this lesson - Recognize the most common plausible bugs in generated quantitative code (arithmetic drawdown, risk-free rate not de-annualized, VWAP not reset per session, wrongly localized time zone) and write the tests that catch them. - Explain the risk of invented dependencies with its published evidence and check every new dependency against a package index before installing it. - Audit a lockfile: exact versions, hashes, source and upload date of every new package, as the course does with uv. - Keep secrets out of an assistant's context and review its changes for keys before the commit. ## Three risks the usual tests do not see Lessons 6.5.1 and 6.5.2 made sure that an assistant's code does what was asked and does not look into the future. Three risks remain that do not depend on the strategy, but on how the code is produced: ```mermaid flowchart TB A[Assistant] --> F[Functions] A --> D[Dependencies] A --> S[Context and diff] F --> F1[Plausible and wrong] D --> D1[Invented package] S --> S1[Leaked key] F1 --> G[Hand-written tests] D1 --> H[Index and lockfile] S1 --> K[Diff scan] ``` These are not theoretical risks. There are published studies with figures, all with models that have since been superseded, so they serve to size the problem, not to measure today's assistants: | Study | What it measured | Result | |---|---|---| | Pearce et al., IEEE S&P 2022 | 1,689 programs generated by GitHub Copilot in 89 security scenarios | Around 40% vulnerable | | Perry et al., ACM CCS 2023 | A user study, with and without an assistant | With the assistant they wrote less secure code and more often believed it was secure | | Spracklen et al., USENIX Security 2025 | 576,000 code samples from 16 models in two languages | Average rate of invented packages of at least 5.2% in commercial models and 21.7% in open ones; 205,474 distinct invented names | | Lanyado, Lasso Security, 2024 | Registered an empty package under a name that models invented | More than 30,000 real downloads in three months | The second row is the one that matters most in practice: the risk is not only that the code is wrong, but that whoever receives it trusts it more. ## Plausible but incorrect code Four functions that an assistant proposes for a metrics library. All of them run, all return reasonable numbers and all have a bug that produces no failure: ```python import numpy as np import pandas as pd def max_drawdown(r): """Maximum drop from a peak, as a fraction.""" curva = r.cumsum() return float((curva - curva.cummax()).min()) def sharpe_anual(r, rf_anual=0.0): """Annualized Sharpe of daily returns.""" exceso = r - rf_anual return float(exceso.mean() / exceso.std() * np.sqrt(252)) def vwap_intradia(op): """Cumulative VWAP of each session (columns ts, precio, volumen; ts in UTC).""" pv = (op["precio"] * op["volumen"]).cumsum() return pv / op["volumen"].cumsum() def a_utc(ts): """New York times without a zone, to UTC.""" return ts.dt.tz_localize("UTC") r = pd.Series([0.10, -0.10, -0.10]) print(f"drawdown {max_drawdown(r):.4f} · " f"Sharpe {sharpe_anual(r):.2f}") ``` ```output drawdown -0.2000 · Sharpe -4.58 ``` The bugs, one by one: the drawdown **adds** simple returns instead of compounding them; the Sharpe subtracts an **annual** risk-free rate from **daily** returns; the VWAP accumulates from the start of the DataFrame and does not reset each session; and the conversion to UTC **declares** that the times are already UTC instead of localizing them in New York and converting them. The drawdown in the cell, −0.2000, looks right. The real drop from 1.10 to 0.891 is 19%. The tests that catch them are written as in lesson 6.5.1: by hand, from the definition, and at the edges. For each function, a case computed by hand and, when one exists, a property that must always hold: ```python def ops(*filas): return pd.DataFrame(filas, columns=["ts", "precio", "volumen"] ).assign(ts=lambda d: pd.to_datetime( d["ts"], utc=True)) TESTS = { "drawdown: three days by hand": lambda f: np.isclose( f["max_drawdown"](pd.Series([0.10, -0.10, -0.10])), 0.891 / 1.1 - 1), "drawdown: negative first day": lambda f: np.isclose( f["max_drawdown"](pd.Series([-0.10, 0.05])), -0.10), "drawdown: between -1 and 0": lambda f: -1 <= f["max_drawdown"]( pd.Series([0.5, -0.6, -0.6])) <= 0, "Sharpe: zero mean excess": lambda f: abs(f["sharpe_anual"]( pd.Series([0.03 / 252 + 0.01, 0.03 / 252 - 0.01] * 50), rf_anual=0.03)) < 1e-9, "VWAP: resets each session": lambda f: np.isclose( f["vwap_intradia"](ops( ("2024-01-02 15:00", 10.0, 100), ("2024-01-02 16:00", 11.0, 100), ("2024-01-03 15:00", 20.0, 50))).iloc[-1], 20.0), "UTC: winter and summer": lambda f: list( f["a_utc"](pd.Series(pd.to_datetime( ["2024-01-02 09:30", "2024-03-11 09:30"]))) .dt.strftime("%H:%M")) == ["14:30", "13:30"], } def pasar(funciones): return {nombre: bool(t(funciones)) for nombre, t in TESTS.items()} propuesta = {"max_drawdown": max_drawdown, "sharpe_anual": sharpe_anual, "vwap_intradia": vwap_intradia, "a_utc": a_utc} for nombre, ok in pasar(propuesta).items(): print(f"{'pass ' if ok else 'FAIL '} {nombre}") ``` ```output FAIL drawdown: three days by hand FAIL drawdown: negative first day FAIL drawdown: between -1 and 0 FAIL Sharpe: zero mean excess FAIL VWAP: resets each session FAIL UTC: winter and summer ``` The four functions fail all six tests. None of them is sophisticated: three days of returns computed by hand, a negative first day, the property that a drawdown lies between −1 and 0 (with +50%, −60% and −60%, the assistant's function gives −120%), two returns symmetric around the daily risk-free rate (zero Sharpe), a second session with a single trade (its VWAP is its price) and two dates on either side of the US daylight saving change of March 10, 2024 (9:30 in New York is 14:30 UTC in January and 13:30 in March). What they have in common is that they are written from the definition, without looking at the code. The first fix of the drawdown that occurs to almost everyone, and to almost every assistant, is to compound: ```python def max_drawdown_v2(r): curva = (1 + r).cumprod() return float((curva / curva.cummax() - 1).min()) v2 = dict(propuesta, max_drawdown=max_drawdown_v2) {k: v for k, v in pasar(v2).items() if k.startswith("drawdown")} ``` ```output {'drawdown: three days by hand': True, 'drawdown: negative first day': False, 'drawdown: between -1 and 0': True} ``` It passes two of the three drawdown tests and fails the one with the negative first day: the curve starts at 0.9 and so does its running maximum, so the drop from the starting capital is not seen. It is a bug inside the fix of another, and in a backtest that starts by losing it underestimates the worst drawdown. The good version adds the starting point (capital 1) before compounding. The four fixed functions pass all six tests: ```python def max_drawdown_ok(r): curva = pd.concat([pd.Series([1.0]), (1 + r).cumprod()], ignore_index=True) return float((curva / curva.cummax() - 1).min()) def sharpe_anual_ok(r, rf_anual=0.0): exceso = r - rf_anual / 252 return float(exceso.mean() / exceso.std() * np.sqrt(252)) def vwap_intradia_ok(op): sesion = (op["ts"].dt.tz_convert("America/New_York") .dt.normalize()) pv = (op["precio"] * op["volumen"]).groupby(sesion).cumsum() return pv / op["volumen"].groupby(sesion).cumsum() def a_utc_ok(ts): return (ts.dt.tz_localize("America/New_York") .dt.tz_convert("UTC")) corregidas = {"max_drawdown": max_drawdown_ok, "sharpe_anual": sharpe_anual_ok, "vwap_intradia": vwap_intradia_ok, "a_utc": a_utc_ok} print(all(pasar(corregidas).values())) ``` ```output True ``` How much do these bugs matter with data? The equal-weighted portfolio of the stocks in the level 2 universe (synthetic data, from 2006 to 2025, with the 2008 crisis), with a risk-free rate of 2% a year: ```python ajustados = pd.read_parquet("universo2_ajustados.parquet") P = ajustados.pivot(index="fecha", columns="ticker", values="cierre_ajustado") cartera = P.pct_change(fill_method=None).mean(axis=1).iloc[1:] for nombre, mal, bien in [ ("worst drawdown", max_drawdown(cartera), max_drawdown_ok(cartera)), ("Sharpe with rf 2%", sharpe_anual(cartera, 0.02), sharpe_anual_ok(cartera, 0.02))]: print(f"{nombre:<18} assistant {mal:8.3f} correct {bien:7.3f}") ``` ```output worst drawdown assistant -0.953 correct -0.631 Sharpe with rf 2% assistant -24.659 correct 0.290 ``` With data, the two bugs stop being details (the figure at the top shows the drawdown over the whole series). The portfolio's worst drawdown with the assistant's function is 95.3%, against a real 63.1%: adding simple returns over a long decline exaggerates the loss (and over long rallies it understates it, as the figure shows between 2010 and 2018). And the Sharpe with a risk-free rate of 2% a year goes from 0.290 to −24.659: subtracting 2% from every daily return is subtracting 504% a year. The second bug is so large that someone would see it; the first, in a report with many figures, not necessarily. There is a fifth kind of bug, the easiest: the **invented API**. An assistant writes a call with a plausible name that the library does not have. It is the good case, because it fails as soon as it runs: ```python raises cartera.rolling(20).zscore() ``` ```output AttributeError: 'Rolling' object has no attribute 'zscore' ``` The danger is not the code that fails, but the code that works. That is the reason for the order of lessons 6.5.1 to 6.5.3: tests are written first and by hand, cases are chosen at the edges (first day, daylight saving change, second session) and properties are always checked (a drawdown cannot go beyond −100%). ## Invented dependencies A model that generates code also generates its `import` statements. If the name it invents sounds right, the person who receives it does the natural thing: `pip install` with that name. If the package does not exist, the command fails and nothing happens. The risk starts when someone registers that name on the public index with malicious code, knowing that models repeat it. Spracklen et al. describe package hallucinations as a persistent, systematic phenomenon of code-generating models, not as a rare accident. It is a variant of *typosquatting* (registering names similar to those of popular packages to exploit typos) that needs no typo: the model supplies the name. There is an older and subtler trap: the name in the `import` is not always the name of the package. `import sklearn` comes from `scikit-learn`; the `sklearn` package on PyPI exists, but its description says “deprecated sklearn package, use scikit-learn instead” (consulted on October 23, 2026). The defense is to check every **new** dependency before installing it. PyPI publishes the metadata of each project in a JSON API (`/pypi//json`, with the description, the versions and the upload date of each file), and a project that does not exist returns 404. The cell starts a **simulated** index with that same response shape, with no network. The six real packages carry their PyPI data from October 23, 2026 (first upload and number of versions); `quantmetrics-ext` and `pandsa` are fictitious, registered in the simulator 6 and 20 days ago. On the real PyPI, on that date, neither of them existed. ```python import json import threading from datetime import date, timedelta from http.server import BaseHTTPRequestHandler, HTTPServer HOY = date(2026, 10, 23) # fixed date: nothing depends on the clock def proyecto(nombre, resumen, primera, versiones): dias = np.linspace(0, (HOY - primera).days, versiones) return {"info": {"name": nombre, "summary": resumen}, "releases": {f"0.{i}": [{"upload_time_iso_8601": f"{primera + timedelta(days=int(d))}" "T12:00:00Z"}] for i, d in enumerate(dias)}} INDICE = {p["info"]["name"]: p for p in [ proyecto("pandas", "Powerful data structures for data " "analysis", date(2009, 12, 25), 120), proyecto("numpy", "Fundamental package for array computing", date(2006, 12, 2), 152), proyecto("scikit-learn", "A set of python modules for machine " "learning", date(2011, 9, 22), 76), proyecto("sklearn", "deprecated sklearn package, use " "scikit-learn instead", date(2015, 7, 15), 10), proyecto("quantstats", "Portfolio analytics for quants", date(2019, 5, 1), 82), proyecto("pyarrow", "Python library for Apache Arrow", date(2017, 3, 16), 66), proyecto("quantmetrics-ext", "Quant metrics extensions", HOY - timedelta(days=6), 1), proyecto("pandsa", "Data analysis toolkit", HOY - timedelta(days=20), 1), ]} class Indice(BaseHTTPRequestHandler): def do_GET(self): partes = self.path.strip("/").split("/") # pypi/

/json datos = INDICE.get(partes[1]) if len(partes) == 3 else None cuerpo = json.dumps(datos).encode() if datos else b"{}" self.send_response(200 if datos else 404) self.send_header("Content-Type", "application/json") self.end_headers() self.wfile.write(cuerpo) def log_message(self, *args): pass servidor = HTTPServer(("127.0.0.1", 0), Indice) threading.Thread(target=servidor.serve_forever, daemon=True).start() URL = f"http://127.0.0.1:{servidor.server_port}" print(len(INDICE), "projects in the simulated index") ``` ```output 8 projects in the simulated index ``` The dependency reviewer queries each name and applies five checks: that it exists, that it is not too new, that it has a history (more than one version), that it is not marked as deprecated and that it does not look too much like the name of a popular package without being it: ```python import difflib import urllib.error import urllib.request POPULARES = ["pandas", "numpy", "scikit-learn", "scipy", "matplotlib", "pyarrow", "polars", "requests"] def consultar(nombre): try: with urllib.request.urlopen(f"{URL}/pypi/{nombre}/json", timeout=5) as r: return json.load(r) except urllib.error.HTTPError as e: if e.code == 404: return None raise def revisar(nombre, dias_minimos=90): datos = consultar(nombre) if datos is None: return "DOES NOT EXIST: do not install (invented name?)" subidas = [f["upload_time_iso_8601"][:10] for fs in datos["releases"].values() for f in fs] edad = (HOY - date.fromisoformat(min(subidas))).days avisos = [] if edad < dias_minimos: avisos.append(f"published {edad} days ago") if len(datos["releases"]) < 2: avisos.append("a single version") if "deprecated" in datos["info"]["summary"].lower(): avisos.append("deprecated: " + datos["info"]["summary"]) parecido = difflib.get_close_matches(nombre, POPULARES, 1, 0.8) if parecido and parecido[0] != nombre: avisos.append(f"resembles {parecido[0]}") return "; ".join(avisos) or f"ok ({edad // 365} years)" PROPUESTA = ["pandas", "numpy", "sklearn", "quantmetrics-ext", "finsignal-toolkit", "pandsa", "pyarrow"] for nombre in PROPUESTA: print(f"{nombre:<18} {revisar(nombre)}") ``` ```output pandas ok (16 years) numpy ok (19 years) sklearn deprecated: deprecated sklearn package, use scikit-learn instead quantmetrics-ext published 6 days ago; a single version finsignal-toolkit DOES NOT EXIST: do not install (invented name?) pandsa published 20 days ago; a single version; resembles pandas pyarrow ok (9 years) ``` *Fig. — The dependencies the assistant proposes in the simulated index: age and number of releases. Established packages sit top right; those registered days ago, with one release, bottom left. finsignal-toolkit is missing: it does not exist.* Log-log scatter. pandas, numpy, scikit-learn, pyarrow and quantstats are 7 to 20 years old with 66 to 152 releases. sklearn, deprecated, 11 years and 10 releases. pandsa, 20 days and 1 release. quantmetrics-ext, highlighted, 6 days and 1 release, left of the 90-day line. The reviewer lets the three established packages through and stops the other four, each for a different reason. `sklearn` exists and is ten years old, but it is marked as deprecated: the assistant wanted `scikit-learn`. `finsignal-toolkit` does not exist: it is the harmless case, as long as nobody registers it. `quantmetrics-ext` is the attack case: a name a model would invent, registered 6 days ago, with a single version. And `pandsa`, registered 20 days ago with one version, resembles `pandas`: it is a typo, whether it comes from a person or from a model. None of those signals proves that a package is malicious, and a new package can be legitimate. What they do is stop the automatic installation until a person looks at where it comes from: its repository, who publishes it, whether the project exists outside the index.

## Lockfiles, hashes and waiting periods Lesson 0.2.2 introduced `uv.lock`: the file that pins the exact version of every package in the environment, its source and the sha256 of each file; it also stores the upload date of each file to the index. It is what makes the course reproducible: the `uv.lock` of the main environment pins more than 160 packages. For security, the lockfile has another virtue: it turns every change of dependencies into a **reviewable diff**. When an assistant adds a package, the new lockfile says exactly what comes in, from where and from when. The cell audits a lockfile change with the shape of `uv.lock` (`[[package]]` tables with `name`, `version`, `source` and files with `hash` and `upload-time`; the hashes are shortened): ```python import tomllib ANTES = """ [[package]] name = "pandas" version = "3.0.6" source = { registry = "https://pypi.org/simple" } wheels = [ { hash = "sha256:9f1c…", upload-time = "2026-09-30T10:01:12Z" }, ] """ DESPUES = ANTES + """ [[package]] name = "quantmetrics-ext" version = "0.0.1" source = { registry = "https://pypi.org/simple" } wheels = [ { hash = "sha256:52ab…", upload-time = "2026-10-17T21:40:03Z" }, ] [[package]] name = "finsignal" version = "1.2.0" source = { git = "https://example.com/u/finsignal.git" } """ def paquetes(texto): return {p["name"]: p for p in tomllib.loads(texto)["package"]} def auditar(antes, despues, espera_dias=7): viejos, nuevos = paquetes(antes), paquetes(despues) for nombre in sorted(set(nuevos) - set(viejos)): p = nuevos[nombre] avisos = [] if p["source"].get("registry") != "https://pypi.org/simple": avisos.append(f"source outside PyPI: {p['source']}") ficheros = p.get("wheels", []) + ( [p["sdist"]] if "sdist" in p else []) if not ficheros or not all("hash" in f for f in ficheros): avisos.append("no hash") for f in ficheros: subida = date.fromisoformat(f["upload-time"][:10]) if (HOY - subida).days < espera_dias: avisos.append(f"uploaded {(HOY - subida).days} " "days ago") print(f"new {nombre} {p['version']}: " f"{'; '.join(avisos) or 'ok'}") auditar(ANTES, DESPUES) ``` ```output new finsignal 1.2.0: source outside PyPI: {'git': 'https://example.com/u/finsignal.git'}; no hash new quantmetrics-ext 0.0.1: uploaded 6 days ago ``` Two new packages and two different warnings. `finsignal` does not come from PyPI but from a git repository, without a hash: what gets installed will depend on what is in that repository on the day of installation. `quantmetrics-ext` comes from PyPI with its hash, but its only file was uploaded 6 days ago. The hash guarantees that everyone installs exactly the same thing; it says nothing about whether that thing is legitimate. That is why the two checks go together. In a repository, those checks translate into a few uv and pip commands and options, all from their documentation: ```bash uv add pandas # resolves and writes pyproject.toml and uv.lock uv sync --locked # in CI: fails if uv.lock is not up to date uv pip compile requirements.in --generate-hashes -o requirements.txt pip install --require-hashes -r requirements.txt ``` And in `pyproject.toml`, a waiting period: uv only considers files uploaded before the given date or age (`exclude-newer`, which accepts durations such as `7 days`), so a version published yesterday does not enter the environment until that period has passed: ```toml [tool.uv] exclude-newer = "7 days" ``` ## Secrets in the context, in the code and in the logs Lesson 4.4.5 showed where to store a broker's credentials, how to redact them in logs and how to scan a repository before the commit. A coding assistant adds new paths through which a key can leak: ```mermaid flowchart TB E[Project .env] --> C[Context] C --> P[Provider] C --> G[Generated code] G --> R[Repository] C --> L[Logs] ``` - **Context**: an assistant reads project files to understand the code. If it reads a `.env`, the key travels to the model provider inside the context. - **Generated code**: if the key was in the context, the model can write it into the code “so that it works”, into a test or into an example. - **Logs**: agent sessions store what they read and what they ran; an `env` or a `print` of the configuration leaves the key in those logs. The tools have rules to exclude files, with nuances worth reading in their documentation (consulted on October 23, 2026). Claude Code's shows how to prevent it from reading `.env` files with deny rules in its configuration (`"deny": ["Read(./.env)", "Read(./.env.*)"]`). Cursor's describes `.cursorignore`, which blocks files for its AI features, but warns that complete protection is not guaranteed and that the agent's terminal and MCP tools cannot block access to those files. The conclusion is that of lesson 4.4.5: **the key must not be in the project tree**, but in the system's or the cloud's secret manager, and whatever is in the tree is treated as if it were going to leak. Two defenses that run. The first filters the context before sending it to a model: it leaves out the files that match secret patterns and redacts in the rest whatever looks like a key. The toy keys start with `FALSA_` (Spanish for “fake”), as in lesson 4.4.5, and are not printed: only their fingerprint. ```python import fnmatch import hashlib import math import re from collections import Counter def falsa(nombre, largo=40): return "FALSA_" + hashlib.sha256(nombre.encode()).hexdigest()[ : largo - 6] def huella(valor): return hashlib.sha256(valor.encode()).hexdigest()[:8] CLAVE = falsa("BROKER_SECRET") REPO = { ".env": f"BROKER_SECRET={CLAVE}\n", "config.py": 'BROKER_URL = "https://api.broker.example"\n' 'BROKER_SECRET = os.environ["BROKER_SECRET"]\n', "senales.py": "def momentum(p, n=20):\n" " return p.pct_change(n)\n", "notas.md": f"tested with the key {CLAVE[:30]}… and it works\n", } EXCLUIR = [".env", ".env.*", "*.pem", "*secret*"] TOKEN = re.compile(r"[A-Za-z0-9_+/=-]{20,}") def entropia(t): n = len(t) return -sum(c / n * math.log2(c / n) for c in Counter(t).values()) def contexto_seguro(repo): partes, informe = [], [] for nombre, texto in sorted(repo.items()): if any(fnmatch.fnmatch(nombre, g) for g in EXCLUIR): informe.append(f"{nombre}: excluded") continue def tachar(m): t = m.group(0) if t.startswith("FALSA_") or entropia(t) >= 3.5: informe.append(f"{nombre}: redacted {huella(t)}") return "[REDACTED]" return t partes.append(f"# {nombre}\n{TOKEN.sub(tachar, texto)}") return "\n".join(partes), informe enviar, informe = contexto_seguro(REPO) print("\n".join(informe)) print("any key in what is sent?", CLAVE[:20] in enviar) ``` ```output .env: excluded notas.md: redacted 5ce46225 any key in what is sent? False ``` The filter leaves out the whole `.env` and redacts the key that someone pasted into `notas.md`, a common leak: notes, drafts and saved notebook outputs. `config.py` goes through unchanged because it reads the key from the environment and does not contain it. What is sent carries no key. The report identifies what was redacted by its fingerprint, as in lesson 4.4.5, so that you can search for where else it appears without writing it again. The second defense reviews what the assistant **returns**: only the lines added in its proposal, which is what would reach the repository. In the example, to “simplify” the configuration, the assistant has replaced the read of the environment variable with the value it saw in the context of another session: ```python import difflib as dl propuesta_config = ('BROKER_URL = "https://api.broker.example"\n' f'BROKER_SECRET = "{CLAVE}" # simplified\n') def secretos_en_diff(antes, despues, nombre): hallazgos = [] for linea in dl.unified_diff(antes.splitlines(), despues.splitlines(), lineterm=""): if not linea.startswith("+") or linea.startswith("+++"): continue for t in TOKEN.findall(linea): if t.startswith("FALSA_") or entropia(t) >= 3.5: hallazgos.append(f"{nombre}: +{len(t)} characters, " f"fingerprint {huella(t)}") return hallazgos print(secretos_en_diff(REPO["config.py"], propuesta_config, "config.py") or "no findings") print(secretos_en_diff(REPO["senales.py"], REPO["senales.py"] + "def media(p, n=20):\n" " return p.rolling(n).mean()\n", "senales.py") or "no findings") ``` ```output ['config.py: +40 characters, fingerprint 3f1154ad'] no findings ``` The diff scanner finds the key in the added line of `config.py` and raises no warnings on the legitimate change to `senales.py`. Looking only at the added lines has two advantages: it is fast and it does not repeat the warnings already accepted for the rest of the repository. If the key did get written into the repository, even if it is deleted later, it has to be rotated: it is still in the history (lesson 4.4.5). ## A checklist for every proposal | Risk | Check | Who does it | |---|---|---| | Plausible and wrong code | Hand-written edge-case tests and properties, written first (lessons 6.5.1 and this one) | Person; CI runs them | | Invented API | Run it: it fails on its own | CI | | Backtest biases | Static reviewer, truncation and placebo (lesson 6.5.2) | CI; the person reads the report | | New dependency | Exists, age, versions, deprecated, similar name; read its repository | Person, with the help of a script | | Lockfile | Diff of `uv.lock`: source, hashes, upload date; `uv sync --locked` | CI | | Freshly published version | Waiting period (`exclude-newer`) | Configuration | | Known vulnerabilities | `pip-audit` or equivalent | CI | | Secrets in the context | Outside the tree; context filter; assistant rules | Configuration and person | | Secrets in the diff | Scan of the added lines before the commit (lesson 4.4.5) | Pre-commit hook and CI | None of these checks needs to trust the assistant, and none asks it to review itself. It is the same rule as for the agents in lesson 6.4.5: the model proposes and a deterministic process decides. ## Exercises 1. An assistant proposes `vol_anual = r.std() * np.sqrt(365)` for stocks that trade about 252 sessions a year. Write a hand-written test that catches it and compute how wrong it is on the level 2 portfolio. **Solution.** ```python def vol_anual_asistente(r): return float(r.std() * np.sqrt(365)) def vol_anual(r, sesiones=252): return float(r.std() * np.sqrt(sesiones)) # by hand: alternating returns of +1% and -1% -> deviation 0.01 r_test = pd.Series([0.01, -0.01] * 126) d = r_test.std() print(f"daily deviation {d:.5f}; expected annual " f"{d * np.sqrt(252):.4f}; assistant " f"{vol_anual_asistente(r_test):.4f}") print(f"portfolio: {vol_anual(cartera):.4f} vs " f"{vol_anual_asistente(cartera):.4f} " f"({vol_anual_asistente(cartera) / vol_anual(cartera) - 1:.1%})") ``` ```output daily deviation 0.01002; expected annual 0.1591; assistant 0.1914 portfolio: 0.2012 vs 0.2422 (20.4%) ``` With alternating returns of +1% and −1%, the daily standard deviation is 0.01002 and the correct annual volatility, 0.1591; the assistant's, 0.1914. On the level 2 portfolio, 0.2012 against 0.2422: 20.4% more, which is $\sqrt{365/252} - 1$. In crypto, which trades every day, 365 would be correct; that is why the number of sessions is a parameter and the test fixes the one for stocks. 2. Many `import` names do not match the package name on PyPI (`sklearn` and `scikit-learn`, `yaml` and `PyYAML`, `cv2` and `opencv-python`). Add to the dependency reviewer a table of equivalences that, before querying the index, translates the `import` name into the package name and flags the change. **Solution.** ```python IMPORT_A_PAQUETE = {"sklearn": "scikit-learn", "yaml": "PyYAML", "cv2": "opencv-python", "bs4": "beautifulsoup4"} def revisar_import(nombre_import): paquete = IMPORT_A_PAQUETE.get(nombre_import, nombre_import) nota = (f" (import {nombre_import} -> package {paquete})" if paquete != nombre_import else "") return f"{paquete}: {revisar(paquete)}{nota}" for n in ["sklearn", "pandas", "quantmetrics"]: print(revisar_import(n)) ``` ```output scikit-learn: ok (15 years) (import sklearn -> package scikit-learn) pandas: ok (16 years) quantmetrics: DOES NOT EXIST: do not install (invented name?) ``` `sklearn` is translated to `scikit-learn` and passes; `pandas` does not change; `quantmetrics` is not in the table and does not exist in the index. The table has to be maintained by hand and is short on purpose: only the known cases. A name that is not in it is queried as is, and if it does not exist, installation stops. What you must never do is “fix” a name that does not exist by searching the index for the closest one and installing it: that is exactly what the attack exploits. 3. The context filter excludes `.env` by name. What happens if the key is in `config.local.json`? Propose two changes: one to the filter and another to the repository. **Solution.** The filter excludes by file name, so `config.local.json` would go through whole. If the key has the shape of the toy ones or a high entropy, redaction by content would catch it anyway; if it is short or readable (a password such as `trading2026`), it would not. Two changes: - In the filter: invert the logic. Instead of a list of exclusions, a **list of inclusions** (`*.py`, `*.md`, `*.toml` of the source code) and everything else out. What has not been chosen to be sent is not sent. - In the repository: the local configuration with secrets should not live in the project tree, but in the secret manager or in a user directory outside the repository (lesson 4.4.5), and `config.local.json` should be in `.gitignore` and in the assistant's deny rules as a second line of defense. ## Self-assessment 1. An assistant computes the drawdown with `curva = r.cumsum()` and `(curva - curva.cummax()).min()`. Why is it dangerous? - A. It is not: the sum of returns and their product are equivalent - B. Because it is slow on long series - C. Because it adds simple returns instead of compounding them: it gives drawdowns in points of cumulative return, not as a percentage of the peak, it can go beyond −100%, and on the course data it exaggerates the portfolio's worst drawdown - D. Because `cummax` does not accept missing values *Answer C: Because it adds simple returns instead of compounding them: it gives drawdowns in points of cumulative return, not as a percentage of the peak, it can go beyond −100%, and on the course data it exaggerates the portfolio's worst drawdown.* With returns of +50%, −60% and −60%, the real drawdown is 84%; the assistant's function gives −120%, which is impossible. The property test “the drawdown is between −1 and 0” catches it instantly. On the equal-weighted level 2 portfolio, the assistant's version gives a much larger worst drawdown than the real one. 2. The fixed version of the drawdown uses `(1 + r).cumprod()` but fails the test with a first day of −10%. Why? - A. Because the curve starts at 0.9 and so does the running maximum: the drop from the starting capital of 1 is not seen; the starting point has to be added - B. Because `cumprod` rounds - C. Because the test is wrong - D. Because negative returns do not compound *Answer A: Because the curve starts at 0.9 and so does the running maximum: the drop from the starting capital of 1 is not seen; the starting point has to be added.* The curve's previous maximum has to include the starting capital. It is a second bug inside the fix of the first, and only a case written by hand for that edge finds it. 3. What are “hallucinated” dependencies and why are they a security risk? - A. Packages with programming bugs - B. Package names that a model invents when generating code: if someone registers that name on PyPI with malicious code, whoever follows the suggestion installs it. Spracklen et al. (2025) measured an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open models - C. Packages that the assistant installs without permission - D. Old versions of real packages *Answer B: Package names that a model invents when generating code: if someone registers that name on PyPI with malicious code, whoever follows the suggestion installs it. Spracklen et al. (2025) measured an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open models.* The package does not exist until someone registers it. In 2024, a researcher registered an empty package under a name that models invented, `huggingface-cli`, and it received more than 30,000 real downloads in three months. 4. An assistant adds `quantmetrics-ext` to the dependencies. It exists in the index, with a single version published 6 days ago. What do you do? - A. Install it: it exists in the index - B. Install it with `--require-hashes` - C. Replace it with the previous version - D. Not install it until you know where it comes from: a very recent package, with one version and a name that matches what a model would invent, is the typical shape of the attack; look for the original project and its repository, or write the function *Answer D: Not install it until you know where it comes from: a very recent package, with one version and a name that matches what a model would invent, is the typical shape of the attack; look for the original project and its repository, or write the function.* Existing in the index says nothing about who published it. Hashes guarantee that you install the same thing that was pinned, not that what was pinned is legitimate. A waiting period (like uv's `exclude-newer`) also avoids installing what was published a few days ago. 5. Is a `.cursorignore` or a read-deny rule enough to keep an assistant from seeing the `.env`? - A. Yes, they are guarantees - B. No: Cursor's documentation says the protection is not guaranteed and that the agent's terminal and MCP tools do not respect the file. The safe approach is for the keys not to be in the working directory and to review the changes before the commit - C. Only if the repository is private - D. Yes, if the `.env` is also in `.gitignore` *Answer B: No: Cursor's documentation says the protection is not guaranteed and that the agent's terminal and MCP tools do not respect the file. The safe approach is for the keys not to be in the working directory and to review the changes before the commit.* Rules reduce the risk, but an agent with a terminal can read any file the user can read. Secrets go outside the project tree (system or cloud secret manager, lesson 4.4.5) and the diff of every proposal is scanned before the commit. ## Key points - Plausible but incorrect code does not fail: a drawdown with `cumsum`, an annual risk-free rate subtracted from daily returns, a VWAP that does not reset each session or a time zone localized as UTC give reasonable, wrong numbers. - The tests that catch them are hand-written edge cases (negative first day, daylight saving change, second session) and properties (the drawdown between −1 and 0). The drawdown fix had a second bug that only a hand-written case saw. - Invented dependencies are a measured risk: an average rate of invented packages of at least 5.2% in commercial models and 21.7% in open ones (Spracklen et al., USENIX Security 2025). An invented name registered later received more than 30,000 downloads in three months. - Every new dependency is checked before installing it: that it exists, its age, its versions, whether it is deprecated and whether it resembles a popular package. Then, a lockfile with exact versions and hashes (`uv.lock`), `--locked` in continuous integration and a waiting period for anything just published. - Secrets are not protected with assistant rules but by taking them out of the project tree; what is sent as context is filtered and the diff of every proposal is scanned before the commit. ## Resources ### Documentation - [Lanyado, B. (2024): Diving Deeper into AI Package Hallucinations. Lasso Security](https://lasso.security/blog/ai-package-hallucinations) · lasso.security - [PyPI: the index's JSON API](https://docs.pypi.org/api/json/) · docs.pypi.org - [uv: settings reference (exclude-newer)](https://docs.astral.sh/uv/reference/settings/) · docs.astral.sh - [pip: secure installs with hashes](https://pip.pypa.io/en/stable/topics/secure-installs/) · pip.pypa.io ### Books - [Spracklen, J. et al. (2025): We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. USENIX Security 2025. arXiv:2406.10279](https://arxiv.org/abs/2406.10279) · arxiv.org - [Pearce, H. et al. (2022): Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. IEEE S&P 2022. arXiv:2108.09293](https://arxiv.org/abs/2108.09293) · arxiv.org - [Perry, N. et al. (2023): Do Users Write More Insecure Code with AI Assistants? ACM CCS 2023. arXiv:2211.03622](https://arxiv.org/abs/2211.03622) · arxiv.org Previous lesson: [6.5.2 · Automated review of backtests and code migrations](https://pythonparatrading.com/en/python-trading-course/level-6/automated-backtest-review-and-code-migration/) Next lesson: [6.6.1 · When data becomes “big”: ticks, order books, options and universes](https://pythonparatrading.com/en/python-trading-course/level-6/when-financial-data-becomes-big-data/)