--- title: "7.3.1 Experiment, model and data tracking: MLflow and DVC" description: "MLflow with SQLite for runs, metrics and registered models; DVC and git to version data, and a run reproduced from its tags." url: "https://pythonparatrading.com/en/python-trading-course/level-7/experiment-model-data-tracking-mlflow-dvc/" lang: "en" image: "https://pythonparatrading.com/og/en/python-trading-course/level-7/experiment-model-data-tracking-mlflow-dvc.png" --- # 7.3.1 · Tracking experiments, models and data: MLflow, DVC and a run reproduced end to end > A number in a research report is worth as much as the ability to reproduce it. This lesson puts the level 5 weekly panel under DVC and the code under git, logs seven runs of a ridge model in MLflow (including one that fails and one with leakage), registers the chosen model with its data version and, after a vendor correction, recovers the exact data of that run and gets the same IC. - Course: [Level 7: Production, MLOps, regulation, career and capstone](https://pythonparatrading.com/en/python-trading-course/level-7/) - Module: 7.3 · MLOps for trading models - Duration: 70 min - Prerequisites: [0.2.5 · Git and GitHub: commits, branches, pull requests and .gitignore](https://pythonparatrading.com/en/python-trading-course/level-0/git-and-github-for-trading/), [2.2.4 · Research journal and experiment tracking against data snooping](https://pythonparatrading.com/en/python-trading-course/level-2/research-journal-experiment-tracking/), [5.4.4 · Backtesting ML signals with costs and Deflated Sharpe](https://pythonparatrading.com/en/python-trading-course/level-5/backtesting-ml-signals-deflated-sharpe/) ## By the end of this lesson - List what is needed to reproduce a result (data, code, parameters, environment, seeds) and where git, DVC and MLflow store each piece. - Version a dataset with DVC in a git repository, push it to a remote and read an earlier version from a commit. - Log runs in MLflow with parameters, version tags, per-step metrics and status, including the ones that fail, and query them with filters. - Register a model with a signature, an input example, tags and an alias, and load it by name to get the same predictions. - Reproduce a past run from its tags and explain why the number of logged runs feeds the multiple-testing correction. ## What you need to be able to reproduce Lesson 2.2.4 built an experiment log in a JSONL file, with the fingerprint of the data and of the code on each line and a hash chain to detect edits. Lesson 5.4.4 logged twenty variants of a model so the Deflated Sharpe could be computed at the end, and its last exercise listed the fields a real log would need. This lesson stores them with the tools used in production, and checks at the end that they are enough to reproduce a result. Reproducing a number takes five things: the exact **data**, the exact **code**, the **parameters**, the **environment** (library versions) and the **seeds**. The environment is pinned by `uv.lock` (lesson 0.2.2) and the Docker image (lesson 7.2.1); seeds are parameters. That leaves three pieces and three tools: git for the code, DVC for the data, which does not fit in git, and MLflow for what ties the two together in each run. ```mermaid flowchart TB G[Code: git] --> E[Run] D[Data: DVC] --> E P[Parameters] --> E E --> M[Metrics] E --> O[Model] O --> R[Registry] ``` The experiment is the cross-sectional ridge from level 5: predict next week's return of 400 synthetic stocks from their weekly features, training up to 2019 and measuring the IC (the weekly rank correlation between prediction and outcome) from 2020 to 2025. The level 5 data has two known traps: `revision_analistas` is a leak, and `sentimiento` is empty for stocks without coverage. None of the runs in this lesson is a good model; what matters is what the log says about each one. ## The dataset and the code First, the experiment's dataset: eight features converted to cross-sectional ranks and the label in excess of each week's mean, in a parquet file inside the folder of a new project. ```python import hashlib from pathlib import Path import numpy as np import pandas as pd COLS = ["ret_1s", "mom_12_1", "vol_63d", "volumen_rel", "valor_bp", "sue", "sentimiento", "revision_analistas"] f = pd.read_parquet("panel5_features.parquet", columns=["fecha", "ticker"] + COLS) e = pd.read_parquet("panel5_etiquetas.parquet", columns=["fecha", "ticker", "ret_sig_semana"]) df = f.merge(e, on=["fecha", "ticker"]).dropna( subset=["ret_sig_semana"]) for c in COLS: # cross-sectional rank in [-0.5, 0.5] df[c] = df.groupby("fecha")[c].rank(pct=True) - 0.5 df["y"] = (df["ret_sig_semana"] - df.groupby("fecha")["ret_sig_semana"].transform("mean")) panel = df[["fecha", "ticker"] + COLS + ["y"]].astype( {c: "float32" for c in COLS + ["y"]}) RUTA = Path("proyecto/datos/panel_semanal.parquet") RUTA.parent.mkdir(parents=True) panel.to_parquet(RUTA, index=False) def md5(ruta): return hashlib.md5(Path(ruta).read_bytes()).hexdigest() print(panel.shape, panel["fecha"].min().date(), panel["fecha"].max().date()) print(f"{RUTA.stat().st_size / 1e6:.1f} MB · md5 {md5(RUTA)}") print(panel["sentimiento"].isna().mean().round(3), "empty in sentimiento") ``` ```output (291600, 11) 2012-01-13 2025-12-26 4.5 MB · md5 36eccd106ec9200919d2c447fffccac2 0.32 empty in sentimiento ``` That is 291,600 rows (400 stocks over 729 weeks with a label), from January 2012 to December 2025, in 4.5 MB. 32% of the `sentimiento` values are empty: the rank of a stock with no data point is also empty. The md5 is the fingerprint of the content, and changing a single value changes it completely. Lesson 2.2.4 computed it by hand; from here on DVC computes and stores it. The training code goes in a project file, not in the cell: what git versions is the file, and the commit identifies exactly the code that produced each number. The weekly IC is the Pearson correlation between the ranks of the prediction and of the label within each week, computed with group aggregates: ```python import sys CODIGO = '''\ """Ridge transversal y su IC semanal fuera de muestra.""" from sklearn.linear_model import Ridge CORTE = "2020-01-01" def ic_semanal(datos, pred): r = datos[["fecha", "y"]].assign(p=pred) r[["p", "y"]] = r.groupby("fecha")[["p", "y"]].rank() g = r.groupby("fecha") cov = (r["p"] * r["y"]).groupby(r["fecha"]).mean() - ( g["p"].mean() * g["y"].mean()) return cov / (g["p"].std(ddof=0) * g["y"].std(ddof=0)) def entrenar(panel, features, alpha): tr = panel[panel["fecha"] < CORTE] te = panel[panel["fecha"] >= CORTE] modelo = Ridge(alpha=alpha).fit(tr[features], tr["y"]) return modelo, ic_semanal(te, modelo.predict(te[features])) ''' Path("proyecto/entrenar.py").write_text(CODIGO) Path("proyecto/.gitignore").write_text("__pycache__/\n") sys.path.insert(0, "proyecto") import entrenar ``` The file's docstring (“cross-sectional ridge and its out-of-sample weekly IC”) and the commit messages further down stay as in the Spanish edition of the course: they are part of what git hashes, and changing them would change the commit identifiers the lesson quotes. ## Versioning the data with DVC A parquet file of several megabytes should not go into git: every version would stay forever in the history of every clone. DVC solves this with a pointer: `dvc add` computes the file's md5, copies it to a local cache (`.dvc/cache`, addressed by that md5), writes a small `.dvc` file with the md5 and the size, and adds the original to `.gitignore`. Git versions the `.dvc` file; `dvc push` uploads the content to a **remote** (here, a local folder; in production, S3, Azure, GCS, SSH…) and `dvc pull` downloads it[^1]. ```mermaid flowchart TB W[panel.parquet] --> C[DVC cache] C --> R[Remote] W --> P[panel.parquet.dvc] P --> G[Git repository] ``` ```python import subprocess def sh(*cmd): r = subprocess.run(cmd, cwd="proyecto", capture_output=True, text=True, check=True) return (r.stdout + r.stderr).strip() sh("git", "init", "-q") sh("dvc", "init", "-q") sh("dvc", "add", "-q", "datos/panel_semanal.parquet") print(sh("dvc", "remote", "add", "-d", "local", "../remoto-dvc")) sh("git", "add", "-A") sh("git", "commit", "-q", "-m", "Panel semanal v1 y entrenar.py") sh("dvc", "push", "-q") COMMIT_V1 = sh("git", "rev-parse", "--short", "HEAD") print("commit", COMMIT_V1) print(Path("proyecto/datos/panel_semanal.parquet.dvc").read_text()) print(sh("git", "ls-files")) remoto = [p for p in Path("remoto-dvc").rglob("*") if p.is_file()] print(len(remoto), "file on the remote:", remoto[0].relative_to("remoto-dvc").parts[:2]) ``` ```output Setting 'local' as a default remote. commit e77070e outs: - md5: 36eccd106ec9200919d2c447fffccac2 size: 4529279 hash: md5 path: panel_semanal.parquet .dvc/.gitignore .dvc/config .dvcignore .gitignore datos/.gitignore datos/panel_semanal.parquet.dvc entrenar.py 1 file on the remote: ('files', 'md5') ``` Only `dvc remote add -d` prints anything. The `.dvc` file is what goes into git: the md5, the size in bytes (4,529,279), the algorithm and the path. `git ls-files` confirms the parquet file is not in the repository: there are the `.dvc` file, the code, the DVC configuration and a `datos/.gitignore` that `dvc add` created to exclude the parquet file. The remote has one file, stored under `files/md5/` with its md5 as the name. Commit `e77070e` identifies both the code and the data version; it comes out the same in every run of the lesson because the content, the author and the date of the commit are fixed. ## An experiment in MLflow MLflow organizes work into **experiments** that contain **runs**. Each run stores parameters (what was chosen), metrics (what was measured, optionally per step), tags (free context), artifacts (files, models) and its status. Without a configured server, the MLflow 3.17 client in the environment creates an SQLite database (`mlflow.db`) in the working folder and stores artifacts in `mlruns/`; the general documentation still describes the `mlruns` file store as the default, but that version no longer uses it without an explicit variable. The grid: three feature sets, two values of `alpha` and one more variant with `sentimiento`. Each run carries the code commit and the data md5 as tags: ```python import os import mlflow mlflow.set_experiment("ridge-semanal") DATOS_MD5 = md5(RUTA) BASE = ["ret_1s", "mom_12_1", "vol_63d", "volumen_rel"] REJILLA = [("precio", BASE, a) for a in (1.0, 1e5)] REJILLA += [("precio+fund", BASE + ["valor_bp", "sue"], a) for a in (1.0, 1e5)] REJILLA += [("precio+fund+rev", BASE + ["valor_bp", "sue", "revision_analistas"], a) for a in (1.0, 1e5)] REJILLA += [("precio+fund+sent", BASE + ["valor_bp", "sue", "sentimiento"], 1.0)] for conjunto, feats, alpha in REJILLA: try: with mlflow.start_run(run_name=f"{conjunto} a={alpha:g}"): mlflow.set_tags({"git_commit": COMMIT_V1, "datos_md5": DATOS_MD5, "datos": "datos/panel_semanal.parquet"}) mlflow.log_params({"conjunto": conjunto, "alpha": alpha, "features": ",".join(feats), "corte": entrenar.CORTE}) modelo, ic = entrenar.entrenar(panel, feats, alpha) for anio, v in ic.groupby(ic.index.year).mean().items(): mlflow.log_metric("ic_anual", v, step=anio) mlflow.log_metrics({ "ic_medio": ic.mean(), "ic_t": ic.mean() / ic.std() * np.sqrt(len(ic))}) except ValueError as err: print(f"{conjunto} a={alpha:g}:", str(err).splitlines()[0]) print([p for p in ("mlflow.db", "mlruns") if os.path.exists(p)]) ``` ```output precio+fund+sent a=1: Input X contains NaN. ['mlflow.db'] ``` Six runs finish and one fails: scikit-learn's ridge does not accept empty values, and `sentimiento` has them. The exception leaves the `with` block, and MLflow closes the run with status `FAILED` before the `except` catches it: the attempt stays logged with its parameters and tags. In the working folder `mlflow.db` appears and `mlruns` does not yet, because no run has stored artifacts. ## Search, compare and count Runs are queried with `mlflow.search_runs`, which returns a DataFrame with one column per parameter, metric and tag, and accepts filters with a syntax similar to SQL. The DataFrame also carries identifiers and times, which change on every run: you select the columns you care about. The feature-set names (`precio`, `precio+fund`…) are the run identifiers and stay as they are. ```python runs = mlflow.search_runs(experiment_names=["ridge-semanal"], order_by=["metrics.ic_medio DESC"]) vista = runs[["tags.mlflow.runName", "status", "metrics.ic_medio", "metrics.ic_t"]].set_axis( ["run", "status", "ic_mean", "t"], axis=1) print(vista.round({"ic_mean": 4, "t": 1}).to_string(index=False)) print("logged trials:", len(runs), "· failed:", int((runs["status"] == "FAILED").sum())) TECHO = 0.029 # oracle IC (level 5 README) sosp = mlflow.search_runs(experiment_names=["ridge-semanal"], filter_string=f"metrics.ic_medio > {TECHO}") extra = {f for fs in sosp["params.features"] for f in fs.split(",")} - set(BASE + ["valor_bp", "sue"]) print("above the ceiling:", len(sosp), "· extra features:", extra) cli = mlflow.MlflowClient() r0 = runs[runs["tags.mlflow.runName"] == "precio+fund a=1"].iloc[0] hist = cli.get_metric_history(r0["run_id"], "ic_anual") print({m.step: round(m.value, 3) for m in hist}) ``` ```output run status ic_mean t precio+fund+rev a=1 FINISHED 0.1763 58.7 precio+fund+rev a=100000 FINISHED 0.1763 58.6 precio+fund a=1 FINISHED 0.0243 4.5 precio+fund a=100000 FINISHED 0.0243 4.5 precio a=100000 FINISHED 0.0213 3.7 precio a=1 FINISHED 0.0212 3.6 precio+fund+sent a=1 FAILED NaN NaN logged trials: 7 · failed: 1 above the ceiling: 2 · extra features: {'revision_analistas'} {2020: 0.018, 2021: 0.018, 2022: 0.017, 2023: 0.001, 2024: 0.051, 2025: 0.04} ``` The first two runs give an IC of 0.176 with a t of 58: six times the IC of the data's oracle (0.029), the maximum achievable out of sample. The `metrics.ic_medio > 0.029` filter finds them, and the difference between their features and those of the others points to `revision_analistas`, the leaky feature from level 5. Without the `features` parameter in the log, that number would have been read as a discovery. The honest ones give 0.021 with the price features and 0.024 adding `valor_bp` and `sue`, with t from 3.6 to 4.5; `alpha` changes almost nothing with more than 160,000 training rows. Seven trials were logged, one of them failed: that 7 is the number of attempts the Deflated Sharpe needs (lessons 2.2.4 and 5.4.4), and it would come out lower if failures were not logged. The figure at the top shows all seven together. Finally, the yearly history of the chosen run shows that the IC is not stable: almost zero in 2023 and 0.05 in 2024. ## What to log in a trading experiment The lesson's grid logs the minimum. A trading experiment that will end up in a report or in production needs more fields, and each one has a natural place: | What | Where | Example | |---|---|---| | Data version | tag | `datos_md5`, the commit with the `.dvc` file | | Code version | tag | `git_commit`; no uncommitted changes when running | | Validation | parameters | cutoff, window, purging and embargo (lessons 5.4.2 and 5.4.3) | | Costs | parameters | cost model and execution assumptions | | Seeds | parameters | `semilla` of the model and of the sampling | | Summary result | metrics | mean IC and t, net Sharpe, turnover | | Result over time | per-step metrics | IC per year, drawdown per year | | Full series | artifact | weekly returns or IC, for the Deflated Sharpe and the PBO | | Decision | tag | “discarded: leakage in `revision_analistas`” | What does not go in: credentials (parameters and tags are readable in plain text, lesson 4.4.5), personal data and whole datasets as artifacts (that is what DVC is for). And a process rule: the log is written from the same code that trains, not by hand afterwards, because what gets written down by hand is what worked. ## From the experiment to the model registry A run stores what was tried; the **model registry** stores what will be used. A registered model has a unique name and numbered versions, each linked to the run that produced it; versions accept tags and **aliases**, named references that can be moved from one version to another, like `models:/MyModel@champion` in the MLflow documentation[^2]. The chosen model is saved inside its run with a signature (the input columns and types, inferred from the example) and registered: ```python feats = r0["params.features"].split(",") modelo, ic = entrenar.entrenar(panel, feats, float(r0["params.alpha"])) X_ej = panel.loc[panel["fecha"] == panel["fecha"].max(), feats].head(3) with mlflow.start_run(run_id=r0["run_id"]): info = mlflow.sklearn.log_model( modelo, name="modelo", input_example=X_ej, registered_model_name="ridge-semanal") version = info.registered_model_version cli.set_model_version_tag("ridge-semanal", version, "datos_md5", DATOS_MD5) cli.set_registered_model_alias("ridge-semanal", "candidato", version) mv = cli.get_model_version_by_alias("ridge-semanal", "candidato") print("version", mv.version, "· from the chosen run:", mv.run_id == r0["run_id"], "· tags:", mv.tags) pf = mlflow.pyfunc.load_model("models:/ridge-semanal@candidato") print(pf.metadata.get_input_schema()) print(np.allclose(pf.predict(X_ej), modelo.predict(X_ej))) ``` ```output Successfully registered model 'ridge-semanal'. Created version '1' of model 'ridge-semanal'. version 1 · from the chosen run: True · tags: {'datos_md5': '36eccd106ec9200919d2c447fffccac2'} ['ret_1s': float (required), 'mom_12_1': float (required), 'vol_63d': float (required), 'volumen_rel': float (required), 'valor_bp': float (required), 'sue': float (required)] True ``` MLflow reports that it created the registered model and its version 1. The `candidato` (candidate) alias points to that version, which links to the chosen run and carries the tag with the data md5. The signature, inferred from the input example, is the six columns with their type: a `predict` call missing any of them raises an exception instead of returning meaningless numbers. And the model loaded by its alias through the generic interface (`pyfunc`), as a service that knows nothing about scikit-learn would do, gives the same predictions as the original. The code that uses it asks for `models:/ridge-semanal@candidato`; when there is a better version, the alias moves and that code does not change. ## A vendor correction Data changes. The vendor announces that the 2023 SUE values were published in the file one week early: each value was actually known one week later. The corrected version (v2) shifts `sue` by one week within 2023, and the workflow is the same as with the code: modify, `dvc add`, commit and `dvc push`. ```python v2 = panel.copy() en_2023 = v2["fecha"].dt.year == 2023 v2.loc[en_2023, "sue"] = (v2[en_2023].groupby("ticker")["sue"] .shift(1).fillna(0)) v2.to_parquet(RUTA, index=False) print(sh("dvc", "status")) sh("dvc", "add", "-q", "datos/panel_semanal.parquet") sh("git", "commit", "-q", "-am", "Panel semanal v2: SUE de 2023") sh("dvc", "push", "-q") print(sh("git", "log", "--oneline")) print("md5 v1", DATOS_MD5[:12], "· md5 v2", md5(RUTA)[:12]) print(sh("dvc", "diff", "HEAD~1")) ``` ```output datos/panel_semanal.parquet.dvc: changed outs: modified: datos/panel_semanal.parquet 3562fe4 Panel semanal v2: SUE de 2023 e77070e Panel semanal v1 y entrenar.py md5 v1 36eccd106ec9 · md5 v2 968bf4c6f9f1 Modified: datos/panel_semanal.parquet files summary: 1 modified ``` `dvc status` detects that the file no longer matches its `.dvc` file. After `dvc add` and the commit, the history has two versions, each with its md5, and `dvc diff HEAD~1` sums up the change: one modified file. Both contents are still in the cache and on the remote; the `.dvc` file of each commit says which is which. ## Reproducing the chosen run Now the proof that the log is useful. Starting **only** from the logged run, with nothing from the notebook's memory: its tags say which commit and which md5; `dvc.api.open` reads the file as it was at that commit, from DVC storage[^3]; its parameters say how to train; and its metric says what should come out. ```python import io import dvc.api run = mlflow.get_run(r0["run_id"]) tags, params = run.data.tags, run.data.params with dvc.api.open("datos/panel_semanal.parquet", repo="proyecto", rev=tags["git_commit"], mode="rb") as fh: crudo = fh.read() print("md5 matches the tag:", hashlib.md5(crudo).hexdigest() == tags["datos_md5"]) datos_v1 = pd.read_parquet(io.BytesIO(crudo)) f_run = params["features"].split(",") _, ic_v1 = entrenar.entrenar(datos_v1, f_run, float(params["alpha"])) _, ic_v2 = entrenar.entrenar(v2, f_run, float(params["alpha"])) print(f"logged {run.data.metrics['ic_medio']:.6f} · " f"reproduced {ic_v1.mean():.6f} · with v2 {ic_v2.mean():.6f}") print("2023:", round(ic_v1[ic_v1.index.year == 2023].mean(), 4), "vs", round(ic_v2[ic_v2.index.year == 2023].mean(), 4)) ``` ```output md5 matches the tag: True logged 0.024276 · reproduced 0.024276 · with v2 0.024311 2023: 0.0013 vs 0.0015 ``` The md5 of what was read matches the tag, and the reproduced IC matches the logged one to the sixth decimal place: 0.024276. With the corrected data, the same code and the same parameters give 0.024311, and the 2023 IC goes from 0.0013 to 0.0015. The difference is small, but enough for a number written with v1 to be unverifiable with v2: without the data version, whoever tries to check that 0.0243 months later will get a different number and will not know why. In a real project, the code is also recovered from the commit (`git worktree add ../rep ` or `git checkout`), and the environment from its `uv.lock`. Here the code did not change between v1 and v2, so the loaded module is enough. To work with the old version in the project folder, the path in the DVC documentation is to recover that commit's `.dvc` file and sync: ```python skip title="Going back to a commit's data version" import subprocess subprocess.run(["git", "checkout", "", "--", "datos/panel_semanal.parquet.dvc"], check=True) subprocess.run(["dvc", "checkout"], check=True) # or dvc pull ``` ## Server, team and alternatives Everything above works in a folder, which is what a single person needs. With a team, MLflow is served as a **tracking server** with a shared database (PostgreSQL, MySQL) and an artifact store (S3, Azure, GCS), and the web interface compares runs and plots metrics. The course environment uses `mlflow-skinny`, the client without server or UI, so these commands are not run: ```python skip title="MLflow server and DVC remote in the cloud" import subprocess # tracking server with the lesson's SQLite database subprocess.run(["mlflow", "server", "--backend-store-uri", "sqlite:///mlflow.db", "--port", "5000"]) # in the training code: # mlflow.set_tracking_uri("http://127.0.0.1:5000") # DVC remote in a bucket (credentials outside the repository) subprocess.run(["dvc", "remote", "add", "-d", "nube", "s3://mi-bucket/dvc"], cwd="proyecto") ``` An MLflow server gives access to every experiment and model: it is published on `127.0.0.1` and reached through an SSH tunnel, like Grafana in lesson 7.2.1. The bucket credentials go in the secrets manager or in the process's environment variables (lesson 4.4.5), never in `.dvc/config`, which goes into git (that is what `.dvc/config.local`, which is not versioned, is for). **Weights & Biases** is the best-known commercial alternative: a cloud service (with a self-hosted installation) with the same model of runs, configuration, per-step metrics and artifacts, plus dashboards to compare them. Its documentation is now published inside CoreWeave's. The basic workflow, according to that documentation[^4]: ```python skip title="The same log in Weights & Biases" import wandb config = {"conjunto": "precio+fund", "alpha": 1.0} with wandb.init(project="ridge-semanal", config=config) as run: for anio, v in ic.groupby(ic.index.year).mean().items(): run.log({"ic_anual": v, "anio": anio}) run.log({"ic_medio": ic.mean()}) ``` | | MLflow | Weights & Biases | DVC | |---|---|---|---| | What it stores | runs, metrics, artifacts, registered models | runs, metrics, artifacts, dashboards | data and model versions (and *pipelines*) | | Where | folder, database or your own server | cloud service | local cache and remote (folder, S3, SSH…) | | License | open source (Apache 2.0) | commercial service | open source (Apache 2.0) | | Fits | model registry and deployment without depending on third parties | teams that want dashboards without maintaining a server | large data that must travel with the code | The choice is secondary. What is not optional is what the cell in the previous section does: from any number in a report you can get to its data, its code and its parameters, and obtain it again. Lesson 7.3.2 starts from this log to retrain, validate and promote models with aliases; lesson 7.3.3 monitors the ones in use, live. ## Exercises 1. Compare the yearly IC history (`ic_anual`) of the `precio a=1` and `precio+fund a=1` runs with `get_metric_history`. In which years does adding `valor_bp` and `sue` contribute anything? **Solution.** ```python def historia(nombre): rid = runs.loc[runs["tags.mlflow.runName"] == nombre, "run_id"].iloc[0] return pd.Series({m.step: m.value for m in cli.get_metric_history(rid, "ic_anual")}) comp = pd.DataFrame({"precio": historia("precio a=1"), "precio+fund": historia("precio+fund a=1")}) comp["difference"] = comp["precio+fund"] - comp["precio"] print(comp.round(4)) ``` ```output precio precio+fund difference 2020 0.0109 0.0175 0.0066 2021 0.0204 0.0182 -0.0022 2022 0.0121 0.0171 0.0051 2023 -0.0023 0.0013 0.0036 2024 0.0492 0.0515 0.0022 2025 0.0372 0.0402 0.0030 ``` Adding `valor_bp` and `sue` improves the IC in five of the six years, with differences from 0.002 to 0.007, and makes it slightly worse in 2021. No yearly difference is large compared with the noise of a weekly IC, so the honest conclusion is the mean's: a small and fairly consistent improvement, which counts as one more trial. 2. Write a gate (`puede_registrarse(run)`) that blocks registering a run if it lacks the `git_commit` or `datos_md5` tag, if its status is not `FINISHED`, if it uses any feature from a blocklist or if its mean IC exceeds the ceiling. Run it over the seven runs. **Solution.** ```python NEGRA = {"revision_analistas"} def puede_registrarse(r): motivos = [] for t in ("git_commit", "datos_md5"): if not isinstance(r.get(f"tags.{t}"), str): motivos.append(f"missing {t}") if r["status"] != "FINISHED": motivos.append("did not finish") usadas = set(str(r["params.features"]).split(",")) if usadas & NEGRA: motivos.append(f"uses {sorted(usadas & NEGRA)}") if r["metrics.ic_medio"] > TECHO: motivos.append("IC above the ceiling") return motivos for _, r in runs.iterrows(): print(f"{r['tags.mlflow.runName']:25}", puede_registrarse(r) or "ok") ``` ```output precio+fund+rev a=1 ["uses ['revision_analistas']", 'IC above the ceiling'] precio+fund+rev a=100000 ["uses ['revision_analistas']", 'IC above the ceiling'] precio+fund a=1 ok precio+fund a=100000 ok precio a=100000 ok precio a=1 ok precio+fund+sent a=1 ['did not finish'] ``` The two leaky runs are rejected for two reasons (the blocklisted feature and the impossible IC), the failed one because it did not finish, and the four honest ones pass. In a real project, the gate goes in the same code that registers, right before `register_model`, so nobody skips it by forgetting. 3. A colleague proposes storing the data in git with Git LFS and the experiments in a spreadsheet. What would the team lose compared with the lesson's combination? **Solution.** Git LFS also stores pointers in git and the content elsewhere, so for versioning large files it is a reasonable alternative; what it does not give you is reading a version from Python by commit (`dvc.api.open`), *pipelines*, or a choice of remotes other than the LFS server. The spreadsheet is the serious problem: nobody records failures in it, or the data md5, or the per-year metric, or the commit, and it is edited by hand without a trace. The lesson's log writes itself, from the same code that trains, including the runs that fail; that is what makes it possible to count the trials for the Deflated Sharpe (lesson 2.2.4) and to reproduce a result months later. And the model registry links each deployed version to the run that produced it, which a spreadsheet does not. [^1]: DVC, “Get Started: Data Versioning” (https://dvc.org/doc/start/data-management/data-versioning): `dvc add` creates the `.dvc` file with the md5, the cache goes in `.dvc/cache/files/md5`, `dvc push`/`dvc pull` work with the remote, and to go back to a version, `git checkout` of the `.dvc` file and `dvc checkout`. Accessed on October 23, 2026. [^2]: MLflow, “Model Registry” (https://mlflow.org/docs/latest/ml/model-registry/): unique name, versions starting at 1, tags, the `models://` URI and aliases like `models:/MyModel@champion`. “Tracking” (https://mlflow.org/docs/latest/ml/tracking/) describes the stores and `mlflow server`. Accessed on October 23, 2026. The default SQLite store and the file store warning are those of mlflow-skinny 3.17.2, checked in the course environment. [^3]: DVC, “dvc.api.open()” (https://dvc.org/doc/api-reference/open): opens a file versioned by DVC in a DVC project; `rev` is the git revision (commit, branch or tag). Accessed on October 23, 2026. [^4]: Weights & Biases, “Experiments overview” (https://docs.wandb.ai/guides/track/): `wandb.init` as a context manager, `run.config`, `run.log` and `run.log_artifact`; the page is served as “CoreWeave Docs”. Deployment options, including *Self-Managed*: https://docs.wandb.ai/platform/hosting/. Accessed on October 23, 2026. ## Self-assessment 1. After `dvc add datos/panel_semanal.parquet` and `git commit`, what does the git repository contain? - A. The whole parquet file, compressed by git - B. Nothing: DVC replaces git - C. A copy of the parquet file in `.git/lfs` - D. The `panel_semanal.parquet.dvc` file with the md5 and the size, and a `.gitignore` that excludes the parquet file; the data goes to the DVC cache and to the remote *Answer D: The `panel_semanal.parquet.dvc` file with the md5 and the size, and a `.gitignore` that excludes the parquet file; the data goes to the DVC cache and to the remote.* DVC leaves a small, readable pointer in git. The content lives in `.dvc/cache` (addressed by its md5) and, after `dvc push`, on the remote. That is why a commit identifies an exact version of the data without putting megabytes into the history. 2. One of the logged runs gives a mean IC of 0.176 with a t of 58. What does the tracking log say? - A. That it is the best model and should be registered - B. That its parameters include `revision_analistas`, the leaky feature from level 5, and that its IC exceeds the oracle's ceiling (0.029): it is impossible out of sample - C. That the alpha was badly chosen - D. Nothing: MLflow does not store the features *Answer B: That its parameters include `revision_analistas`, the leaky feature from level 5, and that its IC exceeds the oracle's ceiling (0.029): it is impossible out of sample.* The log stores which features each run used. A result better than the theoretical ceiling is leakage, not a discovery, and the feature list says which one. Without the log, that number could end up in a report with nobody knowing where it came from. 3. The run with `sentimiento` fails because the feature has empty values. Why is it worth keeping it logged as `FAILED`? - A. Because it counts as a trial: the number of attempts is what the Deflated Sharpe needs, and failures are also decisions made by looking at the data - B. Because MLflow does not allow deleting it - C. Because it takes up little space - D. It is not worth it: failures should be deleted so the report stays clean *Answer A: Because it counts as a trial: the number of attempts is what the Deflated Sharpe needs, and failures are also decisions made by looking at the data.* Every variant tried, whether it works or not, is a look at the data (lesson 2.2.4). If failures disappear, the number of trials is underestimated and the multiple-testing correction falls short. 4. The vendor corrects the data and `dvc add` produces a new md5. How do you recover the data the chosen run was trained on? - A. You cannot: DVC only keeps the latest version - B. By downloading the data from the vendor again - C. With the commit stored in the run's `git_commit` tag: `dvc.api.open(…, rev=commit)` reads that version (or `git checkout` of the `.dvc` file and `dvc checkout`), and its md5 matches the `datos_md5` tag - D. With `mlflow.load_model` *Answer C: With the commit stored in the run's `git_commit` tag: `dvc.api.open(…, rev=commit)` reads that version (or `git checkout` of the `.dvc` file and `dvc checkout`), and its md5 matches the `datos_md5` tag.* The DVC cache and remote keep each version by its md5, and the `.dvc` file of each commit says which one belongs to it. The run's `datos_md5` tag lets you check that what you recovered is the same, byte for byte. 5. What is the `candidato` alias of the registered model for? - A. To encrypt the model - B. So the code that uses it asks for `models:/ridge-semanal@candidato` and not a version number: changing versions means moving the alias, without touching that code - C. To delete old versions - D. So MLflow retrains the model *Answer B: So the code that uses it asks for `models:/ridge-semanal@candidato` and not a version number: changing versions means moving the alias, without touching that code.* An alias is a named reference to a specific version, which can be reassigned. Lesson 7.3.2 uses it for the champion/challenger scheme. ## Key points - Reproducing a number takes five things: the exact data, the exact code, the parameters, the environment and the seeds. Git stores the code, DVC the data (by md5) and MLflow ties the two together in each run. - DVC leaves a `.dvc` file in git with the md5 and the size; the content goes to the cache and to a remote. A commit identifies an exact version of the data, and `dvc.api.open(…, rev=…)` reads it. - Each MLflow run stores parameters, tags (commit, data md5), per-step metrics and status; the ones that fail also count as trials for the Deflated Sharpe. - The log exposes the leakage: the run with `revision_analistas` gives an IC of 0.176, six times the oracle's ceiling (0.029); the honest one, 0.024. - The model registry provides versions, tags and aliases: the code asks for `models:/ridge-semanal@candidato` and gets the same predictions as the scikit-learn model. - After the vendor correction, the data of the chosen run is recovered from its commit, its md5 matches the tag and the IC comes out identical; with the new data, it does not. ## Resources ### Documentation - [MLflow: Tracking (runs, stores and server)](https://mlflow.org/docs/latest/ml/tracking/) · mlflow.org - [MLflow: Model Registry (versions, tags and aliases)](https://mlflow.org/docs/latest/ml/model-registry/) · mlflow.org - [DVC: Get Started, Data Versioning](https://dvc.org/doc/start/data-management/data-versioning) · dvc.org - [DVC: Python API, dvc.api.open](https://dvc.org/doc/api-reference/open) · dvc.org - [Weights & Biases: Experiments overview](https://docs.wandb.ai/guides/track/) · docs.wandb.ai ### Data - [Level 5 data: weekly panel features (Parquet)](https://pythonparatrading.com/curso/datos/nivel-5/panel5_features.parquet) · Download · panel5_features.parquet Previous lesson: [7.2.4 · Security and continuity: 2FA, IP allowlists, backups and failure procedures](https://pythonparatrading.com/en/python-trading-course/level-7/trading-security-and-business-continuity/) Next lesson: [7.3.2 · Scheduled retraining, automatic validation and champion/challenger](https://pythonparatrading.com/en/python-trading-course/level-7/model-retraining-validation-champion-challenger/)