Capstone: common requirements, the panel's rubric and preparing the defense.

What is submitted and how it is assessed

The capstone closes the course with a 40- to 80-hour project: one of the five assignments C1 to C5 (guides 7.6.2 to 7.6.6), developed over about ten weeks and defended before an examining panel. Each project guide describes the assignment, the architecture, the starting data and a backbone that already works with the course data. This one gathers what is the same in all five: how the repository is set up and audited, how trials are logged, what goes into the technical report, how time is allocated, how it is scored and how it is defended.

The common requirements, as set out in the level 7 syllabus, are six: a reproducible repository (uv and Docker), documented data, anti-overfitting validation (walk-forward and CPCV or Deflated Sharpe), a risk report, at least four weeks of paper trading with monitoring and a technical report. None is new: each one was taught in a specific lesson, and the assessment consists of checking that the student applies them together, in their own project and without anyone telling them which cell to run.

flowchart TB
  P[Template] --> A[Auditor green]
  A --> V[Validation]
  V --> D{Stopping criterion}
  D --> PT[Paper 4 weeks]
  PT --> M[Technical report]
  M --> E[Submission]
  E --> T[Panel]
  T --> N[Grade]

The rubric in this guide is the course's own: its own criterion, designed to assess a quantitative project with the same standards that would be required in a professional team, not an external standard or certification. The weights and the mistakes that fail a project are published before starting so that nobody has to guess them.

The repository template

Every project starts from the same tree. The pieces were learned in level 7 (structure and quality in 7.1.1, Docker in 7.2.1, CI in 7.2.2, experiment and data tracking in 7.3.1); here they come together with the capstone's three own additions: the data manifest, the trial log and the folder for the paper trading journal. The cell generates a project's template in a folder and validates each configuration file with its parser: tomllib for the pyproject.toml and yaml.safe_load for compose, CI, pre-commit and the configuration. Docker is not installed in the course environment, so the Dockerfile and the compose file are validated as text; the image is built in CI.

import hashlib
import json
import os
import re
import subprocess
import tomllib
from pathlib import Path

import yaml

IMG_PY = "python:3.14.9-slim-trixie@sha256:b347e2dc…"  # lesson 7.2.1
IMG_UV = "ghcr.io/astral-sh/uv:0.13.5@sha256:32a7bb39…"

PLANTILLA = {
"pyproject.toml": """
[project]
name = "capstone"
version = "0.1.0"
requires-python = ">=3.14,<3.15"
dependencies = ["numpy>=2.5", "pandas>=3.0", "pyyaml>=6.0",
                "pydantic-settings>=2.16"]
[dependency-groups]
dev = ["pytest>=9.1", "ruff>=0.17", "pre-commit>=4.7"]
[tool.ruff]
line-length = 88
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
""",
"uv.lock": """
version = 1
requires-python = ">=3.14, <3.15"
[[package]]
name = "numpy"
version = "2.5.4"
[[package]]
name = "pandas"
version = "3.0.6"
[[package]]
name = "pyyaml"
version = "6.0.3"
[[package]]
name = "pydantic-settings"
version = "2.16.1"
""",
".python-version": "3.14\n",
"Dockerfile": f"""
FROM {IMG_PY} AS builder
COPY --from={IMG_UV} /uv /uvx /bin/
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-install-project --no-dev
FROM {IMG_PY}
RUN useradd --system --uid 10001 capstone
WORKDIR /app
COPY --from=builder /app/.venv /app/.venv
COPY src/ src/
COPY configs/ configs/
ENV PATH="/app/.venv/bin:$PATH" PYTHONPATH=/app/src
USER capstone
CMD ["python", "-m", "capstone.vivo"]
""",
"compose.yaml": """
services:
  sistema:
    build: .
    env_file: .env
    restart: unless-stopped
    volumes: ["./estado:/app/estado"]
  redis:
    image: redis:8.2.2-alpine
""",
"Makefile": """
.PHONY: reproduce test lint
reproduce:
\tuv run --locked python -m capstone.reproducir configs/base.yaml
test:
\tuv run --locked pytest -q
lint:
\tuv run --locked ruff check src tests
""",
".pre-commit-config.yaml": """
repos:
  - repo: local
    hooks:
      - id: ruff
        name: ruff
        entry: ruff check
        language: system
        types: [python]
      - id: registro-solo-anadir
        name: append-only trial log
        entry: python -m capstone.registro --comprobar
        language: system
        files: experimentos/registro_pruebas.jsonl
""",
".github/workflows/ci.yml": """
name: ci
on: [push, pull_request]
jobs:
  reproducir:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v5
      - uses: astral-sh/setup-uv@v7
      - run: make lint test reproduce
      - run: docker build -t capstone .
""",
".gitignore": ".env\n.venv/\nestado/\n__pycache__/\n",
".env.example": "IB_CUENTA_PAPEL=DU0000000\n",
"README.md": "# Capstone\n\nReproduce: `make reproduce`.\n",
"configs/base.yaml": "semilla: 7\ncapital: 5000000\ntau: 0.20\n",
"datos/README.md": "# Data\n\nSource, license, cost.\n",
"datos/precios.csv": "fecha,cierre\n2025-01-02,100.0\n",
"src/capstone/__init__.py": "",
"src/capstone/reproducir.py": """
import numpy as np

def main(semilla):
    rng = np.random.default_rng(semilla)
    return rng.normal()
""",
"src/capstone/registro.py": """
import subprocess
import sys

RUTA = "experimentos/registro_pruebas.jsonl"

def comprobar():
    viejo = subprocess.run(["git", "show", f"HEAD:{RUTA}"],
                           capture_output=True, text=True).stdout
    if not open(RUTA).read().startswith(viejo):
        sys.exit("trial log rewritten: it is append-only")

if __name__ == "__main__" and "--comprobar" in sys.argv:
    comprobar()
""",
"tests/test_reproducir.py": """
from capstone.reproducir import main

def test_semilla():
    assert main(7) == main(7)
""",
"experimentos/registro_pruebas.jsonl": "",
"memoria/memoria.md": "# Technical report\n",
"papel/diario/.gitkeep": "",
}


def crear(raiz, ficheros):
    raiz = Path(raiz)
    for ruta, texto in ficheros.items():
        p = raiz / ruta
        p.parent.mkdir(parents=True, exist_ok=True)
        p.write_text(texto.lstrip("\n"))
    datos = sorted(p for p in (raiz / "datos").iterdir()
                   if p.suffix != ".md" and p.name != "MANIFEST")
    (raiz / "datos/MANIFEST").write_text("".join(
        f"{hashlib.sha256(p.read_bytes()).hexdigest()}  {p.name}\n"
        for p in datos))
    return raiz


raiz = crear("bueno", PLANTILLA)
tomllib.loads((raiz / "pyproject.toml").read_text())
for f in ("compose.yaml", ".pre-commit-config.yaml",
          ".github/workflows/ci.yml", "configs/base.yaml"):
    yaml.safe_load((raiz / f).read_text())
print(sum(p.is_file() for p in raiz.rglob("*")), "files;",
      "folders:",
      " ".join(sorted({p.parts[1] for p in raiz.rglob("*/*")})))

22 files; folders: .github configs datos experimentos memoria papel src tests

The folder and file names of the template (datos/ for data, experimentos/registro_pruebas.jsonl for the trial log, memoria/ for the technical report, papel/diario/ for the paper trading journal) stay in Spanish, like the course data. Each file has a specific job, and almost all of them are justified by what happens without it:

In the example, the uv.lock is an excerpt with the shape of the real one so that the auditor has something to read; in the project it is generated by uv lock and not edited by hand. The uv documentation distinguishes two sync options: with --locked, uv fails if the lockfile does not match the pyproject.toml; with --frozen, it uses the lockfile without checking it. For CI and the image, --locked: an outdated lockfile is exactly what CI has to catch. The images carry their @sha256 digest (abbreviated in the cell), as in lesson 7.2.1: a tag like 3.14.9-slim can move, the digest cannot. If the data are large, the manifest is replaced by DVC (lesson 7.3.1) and the auditor looks for the .dvc files.

A reproducibility auditor

The question the panel asks is simple: does one command, on a clean machine, give the figures in the technical report? The auditor first answers everything that can be checked without running the project. Twelve rules, each with the mistake it prevents. Lesson 7.5.3 had a similar auditor for publishing a portfolio; this one looks at what breaks reproduction and what the rubric penalizes, including the git history:

REG = "experimentos/registro_pruebas.jsonl"
SECRETO = re.compile(
    r"(AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----"
    r"|(?i:api[_-]?key|secret|token|password)\s*[=:]\s*"
    r"['\"]?[A-Za-z0-9/+_\-]{16,})")


def git(raiz, *args):
    return subprocess.run(["git", "-C", str(raiz), *args],
                          capture_output=True, text=True,
                          check=True).stdout


def auditar(raiz):
    raiz, r = Path(raiz), {}
    leer = lambda f: ((raiz / f).read_text()
                      if (raiz / f).exists() else "")
    py = tomllib.loads(leer("pyproject.toml") or "")
    deps = [re.split(r"[<>=!~ \[]", d)[0].lower()
            for d in py.get("project", {}).get("dependencies", [])]
    lock = leer("uv.lock")
    r["R01 lockfile with every dependency"] = bool(lock) and all(
        f'name = "{d}"' in lock for d in deps)
    r["R02 Python version pinned"] = bool(leer(".python-version"))
    df = leer("Dockerfile")
    froms = re.findall(r"^FROM (\S+)", df, re.M)
    r["R03 exact image, uv --locked and non-root"] = (
        bool(froms) and all("@sha256:" in f for f in froms)
        and "uv sync --locked" in df and "\nUSER " in df)
    r["R04 one command: make reproduce"] = bool(
        re.search(r"^reproduce:", leer("Makefile"), re.M))
    codigo = "\n".join(p.read_text()
                       for p in (raiz / "src").rglob("*.py"))
    r["R05 explicit seeds"] = not re.search(
        r"default_rng\(\s*\)|np\.random\.(rand|randn|normal|seed)\(",
        codigo)
    man = dict(l.split()[::-1] for l in
               leer("datos/MANIFEST").splitlines() if l.strip())
    fich = {p.name: hashlib.sha256(p.read_bytes()).hexdigest()
            for p in (raiz / "datos").glob("*")
            if p.suffix != ".md" and p.name != "MANIFEST"}
    r["R06 fingerprinted data"] = bool(fich) and man == fich
    seguidos = git(raiz, "ls-files").split()
    r["R07 .env kept out of git"] = (".env" in leer(".gitignore").split()
                                     and ".env" not in seguidos)
    historia = git(raiz, "log", "-p", "--all", "--format=%H")
    r["R08 no secrets in the history"] = not SECRETO.search(historia)
    versiones = [git(raiz, "show", f"{c}:{REG}") for c in
                 git(raiz, "log", "--format=%H", "--", REG).split()]
    r["R09 append-only trial log"] = all(
        nueva.startswith(vieja) for nueva, vieja
        in zip(versiones, versiones[1:]))
    ci = yaml.safe_load(leer(".github/workflows/ci.yml") or "{}")
    pasos = json.dumps(ci.get("jobs", {}))
    r["R10 CI reproduces and builds"] = ("reproduce" in pasos
                                         and "docker build" in pasos)
    r["R11 tests"] = any((raiz / "tests").glob("test_*.py"))
    r["R12 pre-commit with ruff"] = "ruff" in leer(
        ".pre-commit-config.yaml")
    return r


def commit(raiz, mensaje, fecha):
    git(raiz, "add", "-A")
    subprocess.run(["git", "-C", str(raiz), "commit", "-q", "-m",
                    mensaje], check=True, env={
        **os.environ, "GIT_AUTHOR_DATE": fecha,
        "GIT_COMMITTER_DATE": fecha})


git(raiz, "init", "-q")
commit(raiz, "template", "2026-01-05T10:00:00+00:00")
(raiz / REG).write_text('{"id": "a1", "sharpe": 0.41}\n')
commit(raiz, "first trial", "2026-01-12T10:00:00+00:00")
with open(raiz / REG, "a") as f:
    f.write('{"id": "b2", "sharpe": 0.63}\n')
commit(raiz, "second trial", "2026-01-19T10:00:00+00:00")
res = auditar(raiz)
print(sum(res.values()), "of", len(res), "rules green")
print(git(raiz, "log", "--format=%h %s").strip())

12 of 12 rules green
bc6d8a4 second trial
0c24c0d first trial
267def2 template

The template passes all twelve. The commit dates are set by hand in the subprocess environment, and the author's identity comes from the lesson's environment variables (in the project, the student's): with a fixed date and identity, the commit identifiers are the same on every run. Now, a repository with the defects most often seen in submissions, each one introduced in the commit where it usually appears:

malo = dict(PLANTILLA)
malo["Dockerfile"] = ("FROM python:3.14-slim\nCOPY . /app\n"
                      "RUN pip install -r /app/requirements.txt\n"
                      'CMD ["python", "-m", "capstone.vivo"]\n')
malo["Makefile"] = "test:\n\tpytest -q\n"
malo["src/capstone/reproducir.py"] = (
    "import numpy as np\n\ndef main():\n"
    "    return np.random.default_rng().normal()\n")
malo["pyproject.toml"] = PLANTILLA["pyproject.toml"].replace(
    '"pyyaml>=6.0"', '"pyyaml>=6.0", "scipy>=1.18"')
malo.pop(".python-version")
malo[".github/workflows/ci.yml"] = (
    "name: ci\non: [push]\njobs:\n  t:\n    runs-on: ubuntu-24.04\n"
    "    steps:\n      - run: pytest\n")
malo.pop(".pre-commit-config.yaml")
rm = crear("malo", malo)
git(rm, "init", "-q")
(rm / "configs/broker.py").write_text(
    'API_KEY = "Zx81kQp0LmN4rT7vW2yB"\n')
commit(rm, "broker connection", "2026-01-05T10:00:00+00:00")
(rm / "configs/broker.py").unlink()            # “it's deleted now”
(rm / ".env").write_text("IB_CUENTA_PAPEL=DU1234567\n")
(rm / ".gitignore").write_text("__pycache__/\n")
(rm / REG).write_text('{"id": "a1", "sharpe": 0.41}\n'
                      '{"id": "b2", "sharpe": -0.20}\n')
commit(rm, "trials", "2026-01-12T10:00:00+00:00")
(rm / REG).write_text('{"id": "a1", "sharpe": 0.41}\n')  # dropped
(rm / "datos/precios.csv").write_text(
    "fecha,cierre\n2025-01-02,100.5\n")           # not updated
commit(rm, "cleanup", "2026-01-19T10:00:00+00:00")
for regla, ok in auditar(rm).items():
    print("ok  " if ok else "FAIL", regla)

FAIL R01 lockfile with every dependency
FAIL R02 Python version pinned
FAIL R03 exact image, uv --locked and non-root
FAIL R04 one command: make reproduce
FAIL R05 explicit seeds
FAIL R06 fingerprinted data
FAIL R07 .env kept out of git
FAIL R08 no secrets in the history
FAIL R09 append-only trial log
FAIL R10 CI reproduces and builds
ok   R11 tests
FAIL R12 pre-commit with ruff

Eleven of twelve fail, and each failure has a specific cause. The image uses a tag that moves and a pip install from a file that does not even exist (R03); there is no make reproduce (R04); the seed is not set (R05); scipy was added to the pyproject.toml but the lockfile was not regenerated (R01); the Python version is missing (R02); the prices file changed and the manifest did not (R06); .env is in git (R07). Two deserve separate attention:

• R08, the deleted key. The student deleted configs/broker.py in the second commit, but the key is still in the first one and anyone who clones the repository recovers it with git log -p. The only response is to treat it as compromised: revoke it at the broker, rewrite the history (with git filter-repo, for example) and note it in the technical report. The auditor's pattern is deliberately simple; in the project it is worth also running a specialized detector (gitleaks, for example) in CI. Secrets management is covered in lesson 7.2.4.
• R09, the rewritten log. Trial b2, with a negative Sharpe, disappeared in the last commit. The auditor sees it because each version of the log must start with the previous one. Hiding trials is one of the mistakes that fail a project on their own, and the git history is the evidence.

The auditor does not replace running make reproduce on a clean machine, which is what the panel will do: it checks what can be checked without running anything, in one second, and that is why it is worth running in CI and before each partial submission. What it does not see is just as important: a system date written into the report, a data download in the middle of the process, an iteration order that depends on a set. Those only show up by running twice and comparing the report's fingerprint (guide 7.6.2).

The trial log and the Deflated Sharpe

The trial log is the file that weighs most in the validation grade, because the two numbers the Deflated Sharpe needs come from it (lesson 2.5.4): how many trials were run, N, and the dispersion of their Sharpes. Each line is a trial and carries, at a minimum:

Everything is logged: the variants that never finished, the ones discarded because of a bug, the ones tried in another notebook. The practical rule is that the log is written when the trial is launched, not when the result is seen. To see what is at stake, the level 2 data include a thousand “strategies” with the known answer. A student tries the 20 variants of two families, 40 trials; one of them, e0133, truly has an edge:

import numpy as np
import pandas as pd
from scipy.stats import norm

est = pd.read_parquet("ruido2_estrategias.parquet").pivot(
    index="fecha", columns="estrategia", values="rendimiento")
verdad = pd.read_csv("ruido2_verdad.csv", index_col="estrategia")
probadas = verdad.index[verdad["familia"].isin(["f37", "f16"])]
x = est[probadas]
sr = x.mean() / x.std()                       # daily Sharpe


def psr(sr, umbral, t, asim, curt):           # lesson 2.5.4
    den = np.sqrt(1 - asim * sr + (curt - 1) / 4 * sr**2)
    return norm.cdf((sr - umbral) * np.sqrt(t - 1) / den)


def sr0(n, var_sr):
    g = np.euler_gamma
    return np.sqrt(var_sr) * ((1 - g) * norm.ppf(1 - 1 / n)
                              + g * norm.ppf(1 - 1 / (n * np.e)))


mejor = sr.idxmax()
r, T = x[mejor], len(x)
print(f"{len(probadas)} trials; the best, {mejor}: annual Sharpe "
      f"{sr[mejor] * np.sqrt(252):.2f} (true "
      f"{verdad.at[mejor, 'sharpe_real']})")
for n in (1, 5, 40):
    u = sr0(n, sr.var()) if n > 1 else 0.0
    print(f"declares N = {n:2d}: threshold {u * np.sqrt(252):.2f} annual,"
          f" DSR {psr(sr[mejor], u, T, r.skew(), r.kurt() + 3):.3f}")
segunda = sr.drop(mejor).idxmax()
print(f"the second, {segunda}: annual Sharpe "
      f"{sr[segunda] * np.sqrt(252):.2f} (true "
      f"{verdad.at[segunda, 'sharpe_real']})")

40 trials; the best, e0133: annual Sharpe 1.46 (true 0.8)
declares N =  1: threshold 0.00 annual, DSR 0.998
declares N =  5: threshold 0.51 annual, DSR 0.968
declares N = 40: threshold 0.94 annual, DSR 0.844
the second, e0902: annual Sharpe 0.74 (true 0.0)

The best of the 40 is e0133, precisely the one with an edge, with a measured Sharpe of 1.46 when the true one is 0.8: the best of a group almost always comes out inflated, even when it is good. If the student declares a single trial, their DSR is 0.998; with five, 0.968; with the 40 they actually ran, the expected maximum of 40 trials with no edge rises to an annual Sharpe of 0.94 and the DSR drops to 0.844, below 0.95. With the honest number, the result is not conclusive even though the strategy has an edge: four years of data do not separate it from what 40 trials of noise would give. The second best, e0902, is pure noise with a measured Sharpe of 0.74.

That is the reading the panel expects: the DSR does not punish having tried a lot, it measures how much the data support the result after having tried. Hiding 39 trials would have turned a 0.844 into a 0.998, and that is why the log is append-only (R09), versioned with the code and cited in the technical report with its number of lines. If the variants are highly correlated (all from one family), their dispersion is small and the threshold drops on its own: there is no need to invent an “effective” N to defend yourself, it is enough to declare the real one and show the dispersion. CPCV and the probability of backtest overfitting (lessons 2.5.3 and 2.5.4) use the return matrix of all the trials, which is also kept.

The technical report

The technical report is the document the panel reads before the defense. It is not a project diary or a tutorial: it is a technical document that justifies each decision with the figures from the report that make reproduce regenerates. Its outline is common to the five projects:

The summary is written last and read first. Part 9 is what sets a good technical report apart: stating precisely what the project does not prove (four weeks of paper trading do not estimate a Sharpe; synthetic data validate the code, not the strategy) protects more than any brilliant figure. An automatic check prevents the most frequent oversights: a missing part, or a cited fingerprint that is not the report's.

PARTES = ["Summary", "Assignment and scope", "Data", "Method",
          "Validation", "Architecture", "Risks", "Paper trading",
          "Limitations", "Reproduction"]


def revisar_memoria(texto, informe):
    huella = hashlib.sha256(informe).hexdigest()[:12]
    titulos = re.findall(r"^## (?:\d+\. )?(.+)$", texto, re.M)
    faltan = [p for p in PARTES if p not in titulos]
    citada = re.search(r"report fingerprint: `([0-9a-f]{12})`", texto)
    return {"missing": faltan,
            "fingerprint": "matches" if citada and citada[1] == huella
            else f"does not match (report {huella})"}


informe = json.dumps({"sharpe_neto": 0.837, "n_pruebas": 24},
                     sort_keys=True).encode()
borrador = "\n".join(f"## {i}. {p}\n\nText." for i, p in
                     enumerate(PARTES, 1) if p != "Limitations")
borrador += "\n\nreport fingerprint: `b465af1864ec`\n"
print(revisar_memoria(borrador, informe))

{'missing': ['Limitations'], 'fingerprint': 'does not match (report 3e519e92608e)'}

The example draft lacks the limitations part and cites a fingerprint that is not the report's: the technical report was written with an earlier results report. It is a common mistake when the technical report is drafted during paper trading and the results report is regenerated afterwards (for example, when recomputing the backtest with the measured costs). The check is added to the Makefile as make memoria and to CI.

The risk report, one of the common requirements, is part 7 of the technical report and is written before paper trading, not after. It has three pieces. First, a table of risks specific to the project (not generic ones: “the market can fall” is not a risk of this system; “the crude oil roll falls on a low-liquidity day” is), each with its qualitative probability, its impact in money or in trading days and the defense the system has. Second, the limits with their values and where they come from: notional per order, daily loss, drawdown from the peak, margin used, exposure by sector or by counterparty. Third, the procedures: what shuts the system down, who turns it back on and what is done when the broker, the data or the network fails (lessons 4.2.5, 7.2.4 and 7.3.3). During paper trading, each incident is recorded against that table: if something happened that was not in it, the table was incomplete, and the technical report says so.

Schedule: ten weeks, 40 to 80 hours

The project fits in about ten calendar weeks. The working hours are 40 to 80 depending on the project and the starting point; the four weeks of paper trading are calendar time that runs on its own, while the student handles alerts, reconciles every day and writes the technical report.

The figure at the top draws the same schedule. Three rules make it work:

• Paper trading starts in week 6 or earlier. Four calendar weeks cannot be compressed. If execution is not ready in week 5, what gets cut is scope (fewer markets, one rule fewer), not days of paper trading.
• The decision code is not changed during paper trading. A bug that forces a change is fixed, recorded and restarts the 28-day counter. Changing alert thresholds or fixing the monitoring does not restart it; changing the signal, the size or the limits does.
• The week 4 stopping criterion is respected. If the validation says the system does not pass, paper trading is done anyway (what is assessed is the system, not its profitability), but the technical report says so and does not present the system as tradable.

Four weeks are 28 calendar days and at least 20 sessions of the market the system trades; in crypto (C5), 28 days of continuous trading.

The rubric and the grade calculator

The rubric scores seven criteria from 0 to 10. The grade is the average weighted by the weights in the table, and six mistakes fail a project on their own, whatever the average. Each project guide (7.6.2 to 7.6.6) specifies what evidence each criterion asks for in that project.

Mistakes that fail a project on their own:

1. Demonstrable look-ahead: the result uses information that was not available at the decision date.
2. Results that do not reproduce with the documented command.
3. Backtest without costs.
4. Credentials in the repository or in its history.
5. Hiding the number of trials or selecting the evaluation period after the fact.
6. Fewer than four weeks of paper trading.

The scales for each criterion are indicative: 0-2, absent or wrong; 3-4, present with major mistakes; 5-6, correct in the essentials; 7-8, complete and justified; 9-10, additionally, with something the panel did not expect (an extra check, a sensitivity analysis that changes a decision, a paper trading incident well resolved). The grading follows the usual scale in Spain: suspenso (fail) below 5, aprobado (pass) from 5 to 6.9, notable (very good) from 7 to 8.9 and sobresaliente (outstanding) from 9.

PESOS = {"reproducibilidad": 0.15, "datos": 0.15, "validacion": 0.20,
         "riesgo_ejecucion": 0.15, "paper_trading": 0.15,
         "codigo": 0.10, "memoria_defensa": 0.10}
SUSPENDEN = {
    "anticipacion": "demonstrable look-ahead",
    "no_reproduce": "does not reproduce with the documented command",
    "sin_costes": "backtest without costs",
    "credenciales": "credentials in the repository or the history",
    "pruebas_ocultas": "hidden trials or period chosen after the fact",
    "papel_corto": "fewer than 4 weeks of paper trading"}


def nota(puntos, errores=(), dias_papel=None):
    if set(puntos) != set(PESOS):
        raise ValueError(f"missing: {set(PESOS) - set(puntos)}")
    if not all(0 <= v <= 10 for v in puntos.values()):
        raise ValueError("scores range from 0 to 10")
    errores = set(errores)
    if dias_papel is not None and dias_papel < 28:
        errores.add("papel_corto")
    if raros := errores - set(SUSPENDEN):
        raise ValueError(f"unknown mistake: {raros}")
    n = round(sum(PESOS[k] * v for k, v in puntos.items()), 2)
    if errores:
        return n, "fail", sorted(SUSPENDEN[e] for e in errores)
    cal = ("fail" if n < 5 else "pass" if n < 7
           else "very good" if n < 9 else "outstanding")
    return n, cal, []


assert abs(sum(PESOS.values()) - 1) < 1e-12
A = dict(zip(PESOS, (8, 7, 8, 7, 7, 6, 8)))
B = dict(zip(PESOS, (9, 9, 9, 8, 8, 9, 8)))
C = dict(zip(PESOS, (6, 5, 4, 6, 7, 5, 5)))
print("A:", nota(A, dias_papel=29))
print("B:", nota(B, errores=["credenciales"], dias_papel=30))
print("C:", nota(C, dias_papel=21))
print("C with 28 days:", nota(C, dias_papel=28))

A: (7.35, 'very good', [])
B: (8.6, 'fail', ['credentials in the repository or the history'])
C: (5.4, 'fail', ['fewer than 4 weeks of paper trading'])
C with 28 days: (5.4, 'pass', [])

Student A gets a 7.35, very good: solid in everything, somewhat weak in code quality. B has the best average, 8.6, and fails: there is a credential in the history. C reaches a 5.4 with 21 days of paper trading and fails because of the days; with the 28 days they would have passed with the same average. The calculator always returns the average even when there is a mistake that fails the project, so that the student knows what grade they would get by fixing it in a resit; and it rejects scores outside the scale, missing criteria and mistakes that are not on the list, which is what prevents a rubric from being applied differently to two students.

The figure shows the distribution of weights. Validation weighs more than any other criterion because it is where most quantitative projects fail; reproducibility and data weigh the same as risk, because without them no other figure can be checked.

The defense

The defense is a 20-minute presentation followed by a round of questions from the panel. The 20 minutes are not enough to cover the whole project, and they should not be: the panel has already read the technical report. They serve to show that the student understands their decisions and their limits. A breakdown that works:

Reproducing at the end is not theater: if the fingerprint matches, everything before it is backed up. If the project takes more than a few minutes to reproduce, it is launched at the start of the presentation and checked at the end.

The panel's questions tend to go to the same places, and it pays to have them prepared with figures from the report:

Three tips that come from watching defenses: answer with the figure when the question asks for one (“the measured cost was 0.7 ticks per contract versus 0.5 assumed”); say “I don't know” and how one would find out, instead of improvising; and do not defend a result that your own log contradicts. The panel scores higher someone who acknowledges a limit precisely than someone who presents a high Sharpe without being able to explain how many trials are behind it.

Exercises

1. Add a rule R13 to the auditor that checks the paper trading journal: at least 28 calendar days between the first and the last entry and at least 20 distinct sessions. Test it with an example journal of 19 sessions.

def regla_papel(raiz, dias_min=28, sesiones_min=20):
    fechas = sorted(pd.to_datetime(p.stem) for p in
                    (Path(raiz) / "papel/diario").glob("*.jsonl"))
    if not fechas:
        return False, "no journal"
    dias = (fechas[-1] - fechas[0]).days + 1
    ok = dias >= dias_min and len(fechas) >= sesiones_min
    return ok, f"{dias} days, {len(fechas)} sessions"


sesiones = pd.bdate_range("2026-03-02", periods=19)
for d in sesiones:
    (raiz / f"papel/diario/{d.date()}.jsonl").write_text("{}\n")
print("R13:", regla_papel(raiz))

R13: (False, '25 days, 19 sessions')

2. With the 40 trials of the example, how many would the student have to declare for the DSR of the best one to drop below 0.95? And if they add to the log the 20 variants of another family that they also tried?

umbral_n = next(n for n in range(2, 200)
                if psr(sr[mejor], sr0(n, sr.var()), T, r.skew(),
                       r.kurt() + 3) < 0.95)
print("the DSR drops below 0.95 when declaring", umbral_n, "trials")
mas = verdad.index[verdad["familia"].isin(["f37", "f16", "f02"])]
sr60 = est[mas].mean() / est[mas].std()
r60 = est[sr60.idxmax()]
d60 = psr(sr60.max(), sr0(60, sr60.var()), T, r60.skew(),
          r60.kurt() + 3)
print(f"60 trials: best {sr60.idxmax()}, DSR {d60:.3f}")

the DSR drops below 0.95 when declaring 8 trials
60 trials: best e0133, DSR 0.810

3. A classmate proposes presenting the paper trading of weeks 7 to 10, which were better, and not that of weeks 6 to 9, because “both windows are four weeks long”. What would you tell them?