Lesson 7.3.1 · 70 min
Tracking experiments, models and data: MLflow, DVC and a run reproduced end to end
A number in a research report is worth as much as the ability to reproduce it. This lesson puts the level 5 weekly panel under DVC and the code under git, logs seven runs of a ridge model in MLflow (including one that fails and one with leakage), registers the chosen model with its data version and, after a vendor correction, recovers the exact data of that run and gets the same IC.
- Module
- 7.3 · MLOps for trading models
- Duration
- 70 min
- Prerequisites
- 0.2.5 · Git and GitHub: commits, branches, pull requests and .gitignore, 2.2.4 · Research journal and experiment tracking against data snooping, 5.4.4 · Backtesting ML signals with costs and Deflated Sharpe
- Author
- Jesús Cuesta
- Updated
Level 7 lessons
7.1 · Software engineering for trading
7.2 · Infrastructure and deployment
7.3 · MLOps for trading models
7.4 · Regulation, ethics and taxation
- 7.4.1 MiFID II: clients, suitability and algorithmic trading
- 7.4.2 ESMA and CFDs: leverage, close-out and negative balance
- 7.4.3 PRIIPs, the KID and US ETFs
- 7.4.4 MiCA after the transitional period: checking a provider in the registers
- 7.4.5 Market abuse: spoofing, layering and the bot that imitates them by accident
- 7.4.6 Trading taxes in Spain
- 7.4.7 Ethics: personal data, artificial intelligence and selling signals
7.5 · Quant career
7.6 · Final capstone
- 7.6.1 Capstone: common requirements, the panel's rubric and preparing the defense
- 7.6.2 Capstone C1: a multi-asset futures system, from forecast to paper account
- 7.6.3 Capstone C2: systematic options book, volatility surface and real-time Greeks
- 7.6.4 Capstone C3: stock and ETF portfolio with ML, HRP and Spanish taxation
- 7.6.5 Capstone C4: a research platform with LLMs and alternative data
- 7.6.6 Capstone C5: crypto market making or arbitrage with counterparty control
What you need to be able to reproduce
Lesson 2.2.4 built an experiment log in a JSONL file, with the fingerprint of the data and of the code on each line and a hash chain to detect edits. Lesson 5.4.4 logged twenty variants of a model so the Deflated Sharpe could be computed at the end, and its last exercise listed the fields a real log would need. This lesson stores them with the tools used in production, and checks at the end that they are enough to reproduce a result.
Reproducing a number takes five things: the exact data, the exact code, the parameters, the
environment (library versions) and the seeds. The environment is pinned by uv.lock (lesson 0.2.2) and the
Docker image (lesson 7.2.1); seeds are parameters. That leaves three pieces and three tools: git for the code, DVC for
the data, which does not fit in git, and MLflow for what ties the two together in each run.
Mermaid source
flowchart TB G[Code: git] --> E[Run] D[Data: DVC] --> E P[Parameters] --> E E --> M[Metrics] E --> O[Model] O --> R[Registry]
The experiment is the cross-sectional ridge from level 5: predict next week's return of 400 synthetic stocks from
their weekly features, training up to 2019 and measuring the IC (the weekly rank correlation between prediction and
outcome) from 2020 to 2025. The level 5 data has two known traps: revision_analistas is a leak, and sentimiento is
empty for stocks without coverage. None of the runs in this lesson is a good model; what matters is what the log says
about each one.
The dataset and the code
First, the experiment's dataset: eight features converted to cross-sectional ranks and the label in excess of each week's mean, in a parquet file inside the folder of a new project.
import hashlib
from pathlib import Path
import numpy as np
import pandas as pd
COLS = ["ret_1s", "mom_12_1", "vol_63d", "volumen_rel", "valor_bp",
"sue", "sentimiento", "revision_analistas"]
f = pd.read_parquet("panel5_features.parquet",
columns=["fecha", "ticker"] + COLS)
e = pd.read_parquet("panel5_etiquetas.parquet",
columns=["fecha", "ticker", "ret_sig_semana"])
df = f.merge(e, on=["fecha", "ticker"]).dropna(
subset=["ret_sig_semana"])
for c in COLS: # cross-sectional rank in [-0.5, 0.5]
df[c] = df.groupby("fecha")[c].rank(pct=True) - 0.5
df["y"] = (df["ret_sig_semana"]
- df.groupby("fecha")["ret_sig_semana"].transform("mean"))
panel = df[["fecha", "ticker"] + COLS + ["y"]].astype(
{c: "float32" for c in COLS + ["y"]})
RUTA = Path("proyecto/datos/panel_semanal.parquet")
RUTA.parent.mkdir(parents=True)
panel.to_parquet(RUTA, index=False)
def md5(ruta):
return hashlib.md5(Path(ruta).read_bytes()).hexdigest()
print(panel.shape, panel["fecha"].min().date(),
panel["fecha"].max().date())
print(f"{RUTA.stat().st_size / 1e6:.1f} MB · md5 {md5(RUTA)}")
print(panel["sentimiento"].isna().mean().round(3),
"empty in sentimiento")(291600, 11) 2012-01-13 2025-12-26
4.5 MB · md5 36eccd106ec9200919d2c447fffccac2
0.32 empty in sentimientoThat is 291,600 rows (400 stocks over 729 weeks with a label), from January 2012 to December 2025, in 4.5 MB. 32% of
the sentimiento values are empty: the rank of a stock with no data point is also empty. The md5 is the fingerprint
of the content, and changing a single value changes it completely. Lesson 2.2.4 computed it by hand; from here on DVC
computes and stores it.
The training code goes in a project file, not in the cell: what git versions is the file, and the commit identifies exactly the code that produced each number. The weekly IC is the Pearson correlation between the ranks of the prediction and of the label within each week, computed with group aggregates:
import sys
CODIGO = '''\
"""Ridge transversal y su IC semanal fuera de muestra."""
from sklearn.linear_model import Ridge
CORTE = "2020-01-01"
def ic_semanal(datos, pred):
r = datos[["fecha", "y"]].assign(p=pred)
r[["p", "y"]] = r.groupby("fecha")[["p", "y"]].rank()
g = r.groupby("fecha")
cov = (r["p"] * r["y"]).groupby(r["fecha"]).mean() - (
g["p"].mean() * g["y"].mean())
return cov / (g["p"].std(ddof=0) * g["y"].std(ddof=0))
def entrenar(panel, features, alpha):
tr = panel[panel["fecha"] < CORTE]
te = panel[panel["fecha"] >= CORTE]
modelo = Ridge(alpha=alpha).fit(tr[features], tr["y"])
return modelo, ic_semanal(te, modelo.predict(te[features]))
'''
Path("proyecto/entrenar.py").write_text(CODIGO)
Path("proyecto/.gitignore").write_text("__pycache__/\n")
sys.path.insert(0, "proyecto")
import entrenarThe file's docstring (“cross-sectional ridge and its out-of-sample weekly IC”) and the commit messages further down stay as in the Spanish edition of the course: they are part of what git hashes, and changing them would change the commit identifiers the lesson quotes.
Versioning the data with DVC
A parquet file of several megabytes should not go into git: every version would stay forever in the history of every
clone. DVC solves this with a pointer: dvc add computes the file's md5, copies it to a local cache (.dvc/cache,
addressed by that md5), writes a small .dvc file with the md5 and the size, and adds the original to .gitignore.
Git versions the .dvc file; dvc push uploads the content to a remote (here, a local folder; in production, S3,
Azure, GCS, SSH…) and dvc pull downloads it1Note 1 · DVC, “Get Started: Data Versioning” (https://dvc.org/doc/start/data-management/data-versioning): dvc add
creates the .dvc file with the md5, the cache goes in .dvc/cache/files/md5, dvc push/dvc pull work with the
remote, and to go back to a version, git checkout of the .dvc file and dvc checkout. Accessed on October 23,
2026..
Mermaid source
flowchart TB W[panel.parquet] --> C[DVC cache] C --> R[Remote] W --> P[panel.parquet.dvc] P --> G[Git repository]
import subprocess
def sh(*cmd):
r = subprocess.run(cmd, cwd="proyecto", capture_output=True,
text=True, check=True)
return (r.stdout + r.stderr).strip()
sh("git", "init", "-q")
sh("dvc", "init", "-q")
sh("dvc", "add", "-q", "datos/panel_semanal.parquet")
print(sh("dvc", "remote", "add", "-d", "local", "../remoto-dvc"))
sh("git", "add", "-A")
sh("git", "commit", "-q", "-m", "Panel semanal v1 y entrenar.py")
sh("dvc", "push", "-q")
COMMIT_V1 = sh("git", "rev-parse", "--short", "HEAD")
print("commit", COMMIT_V1)
print(Path("proyecto/datos/panel_semanal.parquet.dvc").read_text())
print(sh("git", "ls-files"))
remoto = [p for p in Path("remoto-dvc").rglob("*") if p.is_file()]
print(len(remoto), "file on the remote:",
remoto[0].relative_to("remoto-dvc").parts[:2])Setting 'local' as a default remote.
commit e77070e
outs:
- md5: 36eccd106ec9200919d2c447fffccac2
size: 4529279
hash: md5
path: panel_semanal.parquet
.dvc/.gitignore
.dvc/config
.dvcignore
.gitignore
datos/.gitignore
datos/panel_semanal.parquet.dvc
entrenar.py
1 file on the remote: ('files', 'md5')Only dvc remote add -d prints anything. The .dvc file is what goes into git: the md5, the size in bytes
(4,529,279), the algorithm and the path. git ls-files confirms the parquet file is not in the repository: there are
the .dvc file, the code, the DVC configuration and a datos/.gitignore that dvc add created to exclude the
parquet file. The remote has one file, stored under files/md5/ with its md5 as the name. Commit e77070e
identifies both the code and the data version; it comes out the same in every run of the lesson because the content,
the author and the date of the commit are fixed.
An experiment in MLflow
MLflow organizes work into experiments that contain runs. Each run stores parameters (what was chosen),
metrics (what was measured, optionally per step), tags (free context), artifacts (files, models) and its status.
Without a configured server, the MLflow 3.17 client in the environment creates an SQLite database (mlflow.db) in
the working folder and stores artifacts in mlruns/; the general documentation still describes the mlruns file
store as the default, but that version no longer uses it without an explicit variable. The grid: three feature sets,
two values of alpha and one more variant with sentimiento. Each run carries the code commit and the data md5 as
tags:
import os
import mlflow
mlflow.set_experiment("ridge-semanal")
DATOS_MD5 = md5(RUTA)
BASE = ["ret_1s", "mom_12_1", "vol_63d", "volumen_rel"]
REJILLA = [("precio", BASE, a) for a in (1.0, 1e5)]
REJILLA += [("precio+fund", BASE + ["valor_bp", "sue"], a)
for a in (1.0, 1e5)]
REJILLA += [("precio+fund+rev", BASE + ["valor_bp", "sue",
"revision_analistas"], a) for a in (1.0, 1e5)]
REJILLA += [("precio+fund+sent", BASE + ["valor_bp", "sue",
"sentimiento"], 1.0)]
for conjunto, feats, alpha in REJILLA:
try:
with mlflow.start_run(run_name=f"{conjunto} a={alpha:g}"):
mlflow.set_tags({"git_commit": COMMIT_V1,
"datos_md5": DATOS_MD5,
"datos": "datos/panel_semanal.parquet"})
mlflow.log_params({"conjunto": conjunto, "alpha": alpha,
"features": ",".join(feats),
"corte": entrenar.CORTE})
modelo, ic = entrenar.entrenar(panel, feats, alpha)
for anio, v in ic.groupby(ic.index.year).mean().items():
mlflow.log_metric("ic_anual", v, step=anio)
mlflow.log_metrics({
"ic_medio": ic.mean(),
"ic_t": ic.mean() / ic.std() * np.sqrt(len(ic))})
except ValueError as err:
print(f"{conjunto} a={alpha:g}:", str(err).splitlines()[0])
print([p for p in ("mlflow.db", "mlruns") if os.path.exists(p)])precio+fund+sent a=1: Input X contains NaN.
['mlflow.db']Six runs finish and one fails: scikit-learn's ridge does not accept empty values, and sentimiento has them. The
exception leaves the with block, and MLflow closes the run with status FAILED before the except catches it: the
attempt stays logged with its parameters and tags. In the working folder mlflow.db appears and mlruns does not yet,
because no run has stored artifacts.
Search, compare and count
Runs are queried with mlflow.search_runs, which returns a DataFrame with one column per parameter, metric and tag,
and accepts filters with a syntax similar to SQL. The DataFrame also carries identifiers and times, which change on
every run: you select the columns you care about. The feature-set names (precio, precio+fund…) are the run
identifiers and stay as they are.
runs = mlflow.search_runs(experiment_names=["ridge-semanal"],
order_by=["metrics.ic_medio DESC"])
vista = runs[["tags.mlflow.runName", "status", "metrics.ic_medio",
"metrics.ic_t"]].set_axis(
["run", "status", "ic_mean", "t"], axis=1)
print(vista.round({"ic_mean": 4, "t": 1}).to_string(index=False))
print("logged trials:", len(runs), "· failed:",
int((runs["status"] == "FAILED").sum()))
TECHO = 0.029 # oracle IC (level 5 README)
sosp = mlflow.search_runs(experiment_names=["ridge-semanal"],
filter_string=f"metrics.ic_medio > {TECHO}")
extra = {f for fs in sosp["params.features"]
for f in fs.split(",")} - set(BASE + ["valor_bp", "sue"])
print("above the ceiling:", len(sosp), "· extra features:", extra)
cli = mlflow.MlflowClient()
r0 = runs[runs["tags.mlflow.runName"] == "precio+fund a=1"].iloc[0]
hist = cli.get_metric_history(r0["run_id"], "ic_anual")
print({m.step: round(m.value, 3) for m in hist}) run status ic_mean t
precio+fund+rev a=1 FINISHED 0.1763 58.7
precio+fund+rev a=100000 FINISHED 0.1763 58.6
precio+fund a=1 FINISHED 0.0243 4.5
precio+fund a=100000 FINISHED 0.0243 4.5
precio a=100000 FINISHED 0.0213 3.7
precio a=1 FINISHED 0.0212 3.6
precio+fund+sent a=1 FAILED NaN NaN
logged trials: 7 · failed: 1
above the ceiling: 2 · extra features: {'revision_analistas'}
{2020: 0.018, 2021: 0.018, 2022: 0.017, 2023: 0.001, 2024: 0.051, 2025: 0.04}The first two runs give an IC of 0.176 with a t of 58: six times the IC of the data's oracle (0.029), the maximum
achievable out of sample. The metrics.ic_medio > 0.029 filter finds them, and the difference between their features
and those of the others points to revision_analistas, the leaky feature from level 5. Without the features
parameter in the log, that number would have been read as a discovery. The honest ones give 0.021 with the price
features and 0.024 adding valor_bp and sue, with t from 3.6 to 4.5; alpha changes almost nothing with more than
160,000 training rows. Seven trials were logged, one of them failed: that 7 is the number of attempts the Deflated
Sharpe needs (lessons 2.2.4 and 5.4.4), and it would come out lower if failures were not logged. The figure at the top
shows all seven together. Finally, the yearly history of the chosen run shows that the IC is not stable: almost zero
in 2023 and 0.05 in 2024.
What to log in a trading experiment
The lesson's grid logs the minimum. A trading experiment that will end up in a report or in production needs more fields, and each one has a natural place:
| What | Where | Example |
|---|---|---|
| Data version | tag | datos_md5, the commit with the .dvc file |
| Code version | tag | git_commit; no uncommitted changes when running |
| Validation | parameters | cutoff, window, purging and embargo (lessons 5.4.2 and 5.4.3) |
| Costs | parameters | cost model and execution assumptions |
| Seeds | parameters | semilla of the model and of the sampling |
| Summary result | metrics | mean IC and t, net Sharpe, turnover |
| Result over time | per-step metrics | IC per year, drawdown per year |
| Full series | artifact | weekly returns or IC, for the Deflated Sharpe and the PBO |
| Decision | tag | “discarded: leakage in revision_analistas” |
What does not go in: credentials (parameters and tags are readable in plain text, lesson 4.4.5), personal data and whole datasets as artifacts (that is what DVC is for). And a process rule: the log is written from the same code that trains, not by hand afterwards, because what gets written down by hand is what worked.
From the experiment to the model registry
A run stores what was tried; the model registry stores what will be used. A registered model has a unique name
and numbered versions, each linked to the run that produced it; versions accept tags and aliases, named
references that can be moved from one version to another, like models:/MyModel@champion in the MLflow
documentation2Note 2 · MLflow, “Model Registry” (https://mlflow.org/docs/latest/ml/model-registry/): unique name, versions starting at
1, tags, the models:/<name>/<version> URI and aliases like models:/MyModel@champion. “Tracking”
(https://mlflow.org/docs/latest/ml/tracking/) describes the stores and mlflow server. Accessed on October 23,
2026. The default SQLite store and the file store warning are those of mlflow-skinny 3.17.2, checked in the course
environment.. The chosen model is saved inside its run with a signature (the input columns and types, inferred
from the example) and registered:
feats = r0["params.features"].split(",")
modelo, ic = entrenar.entrenar(panel, feats, float(r0["params.alpha"]))
X_ej = panel.loc[panel["fecha"] == panel["fecha"].max(), feats].head(3)
with mlflow.start_run(run_id=r0["run_id"]):
info = mlflow.sklearn.log_model(
modelo, name="modelo", input_example=X_ej,
registered_model_name="ridge-semanal")
version = info.registered_model_version
cli.set_model_version_tag("ridge-semanal", version, "datos_md5",
DATOS_MD5)
cli.set_registered_model_alias("ridge-semanal", "candidato", version)
mv = cli.get_model_version_by_alias("ridge-semanal", "candidato")
print("version", mv.version, "· from the chosen run:",
mv.run_id == r0["run_id"], "· tags:", mv.tags)
pf = mlflow.pyfunc.load_model("models:/ridge-semanal@candidato")
print(pf.metadata.get_input_schema())
print(np.allclose(pf.predict(X_ej), modelo.predict(X_ej)))Successfully registered model 'ridge-semanal'.
Created version '1' of model 'ridge-semanal'.
version 1 · from the chosen run: True · tags: {'datos_md5': '36eccd106ec9200919d2c447fffccac2'}
['ret_1s': float (required), 'mom_12_1': float (required), 'vol_63d': float (required), 'volumen_rel': float (required), 'valor_bp': float (required), 'sue': float (required)]
TrueMLflow reports that it created the registered model and its version 1. The candidato (candidate) alias points to
that version, which links to the chosen run and carries the tag with the data md5. The signature, inferred from the
input example, is the six columns with their type: a predict call missing any of them raises an exception instead
of returning meaningless numbers. And the model loaded by its alias through the generic interface (pyfunc), as a
service that knows nothing about scikit-learn would do, gives the same predictions as the original. The code that
uses it asks for models:/ridge-semanal@candidato; when there is a better version, the alias moves and that code does
not change.
A vendor correction
Data changes. The vendor announces that the 2023 SUE values were published in the file one week early: each value was
actually known one week later. The corrected version (v2) shifts sue by one week within 2023, and the workflow is the
same as with the code: modify, dvc add, commit and dvc push.
v2 = panel.copy()
en_2023 = v2["fecha"].dt.year == 2023
v2.loc[en_2023, "sue"] = (v2[en_2023].groupby("ticker")["sue"]
.shift(1).fillna(0))
v2.to_parquet(RUTA, index=False)
print(sh("dvc", "status"))
sh("dvc", "add", "-q", "datos/panel_semanal.parquet")
sh("git", "commit", "-q", "-am", "Panel semanal v2: SUE de 2023")
sh("dvc", "push", "-q")
print(sh("git", "log", "--oneline"))
print("md5 v1", DATOS_MD5[:12], "· md5 v2", md5(RUTA)[:12])
print(sh("dvc", "diff", "HEAD~1"))datos/panel_semanal.parquet.dvc:
changed outs:
modified: datos/panel_semanal.parquet
3562fe4 Panel semanal v2: SUE de 2023
e77070e Panel semanal v1 y entrenar.py
md5 v1 36eccd106ec9 · md5 v2 968bf4c6f9f1
Modified:
datos/panel_semanal.parquet
files summary: 1 modifieddvc status detects that the file no longer matches its .dvc file. After dvc add and the commit, the history has
two versions, each with its md5, and dvc diff HEAD~1 sums up the change: one modified file. Both contents are still
in the cache and on the remote; the .dvc file of each commit says which is which.
Reproducing the chosen run
Now the proof that the log is useful. Starting only from the logged run, with nothing from the notebook's memory:
its tags say which commit and which md5; dvc.api.open reads the file as it was at that commit, from DVC storage3Note 3 · DVC, “dvc.api.open()” (https://dvc.org/doc/api-reference/open): opens a file versioned by DVC in a DVC project;
rev is the git revision (commit, branch or tag). Accessed on October 23, 2026.;
its parameters say how to train; and its metric says what should come out.
import io
import dvc.api
run = mlflow.get_run(r0["run_id"])
tags, params = run.data.tags, run.data.params
with dvc.api.open("datos/panel_semanal.parquet", repo="proyecto",
rev=tags["git_commit"], mode="rb") as fh:
crudo = fh.read()
print("md5 matches the tag:",
hashlib.md5(crudo).hexdigest() == tags["datos_md5"])
datos_v1 = pd.read_parquet(io.BytesIO(crudo))
f_run = params["features"].split(",")
_, ic_v1 = entrenar.entrenar(datos_v1, f_run, float(params["alpha"]))
_, ic_v2 = entrenar.entrenar(v2, f_run, float(params["alpha"]))
print(f"logged {run.data.metrics['ic_medio']:.6f} · "
f"reproduced {ic_v1.mean():.6f} · with v2 {ic_v2.mean():.6f}")
print("2023:", round(ic_v1[ic_v1.index.year == 2023].mean(), 4),
"vs", round(ic_v2[ic_v2.index.year == 2023].mean(), 4))md5 matches the tag: True
logged 0.024276 · reproduced 0.024276 · with v2 0.024311
2023: 0.0013 vs 0.0015The md5 of what was read matches the tag, and the reproduced IC matches the logged one to the sixth decimal place: 0.024276. With the corrected data, the same code and the same parameters give 0.024311, and the 2023 IC goes from 0.0013 to 0.0015. The difference is small, but enough for a number written with v1 to be unverifiable with v2: without the data version, whoever tries to check that 0.0243 months later will get a different number and will not know why.
In a real project, the code is also recovered from the commit (git worktree add ../rep <commit> or git checkout),
and the environment from its uv.lock. Here the code did not change between v1 and v2, so the loaded module is
enough. To work with the old version in the project folder, the path in the DVC documentation is to recover that
commit's .dvc file and sync:
import subprocess
subprocess.run(["git", "checkout", "<commit>", "--",
"datos/panel_semanal.parquet.dvc"], check=True)
subprocess.run(["dvc", "checkout"], check=True) # or dvc pullServer, team and alternatives
Everything above works in a folder, which is what a single person needs. With a team, MLflow is served as a tracking
server with a shared database (PostgreSQL, MySQL) and an artifact store (S3, Azure, GCS), and the web interface
compares runs and plots metrics. The course environment uses mlflow-skinny, the client without server or UI, so
these commands are not run:
import subprocess
# tracking server with the lesson's SQLite database
subprocess.run(["mlflow", "server", "--backend-store-uri",
"sqlite:///mlflow.db", "--port", "5000"])
# in the training code:
# mlflow.set_tracking_uri("http://127.0.0.1:5000")
# DVC remote in a bucket (credentials outside the repository)
subprocess.run(["dvc", "remote", "add", "-d", "nube",
"s3://mi-bucket/dvc"], cwd="proyecto")An MLflow server gives access to every experiment and model: it is published on 127.0.0.1 and reached through an SSH
tunnel, like Grafana in lesson 7.2.1. The bucket credentials go in the secrets manager or in the process's environment
variables (lesson 4.4.5), never in .dvc/config, which goes into git (that is what .dvc/config.local, which is not
versioned, is for).
Weights & Biases is the best-known commercial alternative: a cloud service (with a self-hosted installation) with
the same model of runs, configuration, per-step metrics and artifacts, plus dashboards to compare them. Its
documentation is now published inside CoreWeave's. The basic workflow, according to that documentation4Note 4 · Weights & Biases, “Experiments overview” (https://docs.wandb.ai/guides/track/): wandb.init as a context
manager, run.config, run.log and run.log_artifact; the page is served as “CoreWeave Docs”. Deployment
options, including Self-Managed: https://docs.wandb.ai/platform/hosting/. Accessed on October 23, 2026.:
import wandb
config = {"conjunto": "precio+fund", "alpha": 1.0}
with wandb.init(project="ridge-semanal", config=config) as run:
for anio, v in ic.groupby(ic.index.year).mean().items():
run.log({"ic_anual": v, "anio": anio})
run.log({"ic_medio": ic.mean()})| MLflow | Weights & Biases | DVC | |
|---|---|---|---|
| What it stores | runs, metrics, artifacts, registered models | runs, metrics, artifacts, dashboards | data and model versions (and pipelines) |
| Where | folder, database or your own server | cloud service | local cache and remote (folder, S3, SSH…) |
| License | open source (Apache 2.0) | commercial service | open source (Apache 2.0) |
| Fits | model registry and deployment without depending on third parties | teams that want dashboards without maintaining a server | large data that must travel with the code |
The choice is secondary. What is not optional is what the cell in the previous section does: from any number in a report you can get to its data, its code and its parameters, and obtain it again. Lesson 7.3.2 starts from this log to retrain, validate and promote models with aliases; lesson 7.3.3 monitors the ones in use, live.
Exercises
- Compare the yearly IC history (
ic_anual) of theprecio a=1andprecio+fund a=1runs withget_metric_history. In which years does addingvalor_bpandsuecontribute anything?
Solution
def historia(nombre):
rid = runs.loc[runs["tags.mlflow.runName"] == nombre,
"run_id"].iloc[0]
return pd.Series({m.step: m.value for m in
cli.get_metric_history(rid, "ic_anual")})
comp = pd.DataFrame({"precio": historia("precio a=1"),
"precio+fund": historia("precio+fund a=1")})
comp["difference"] = comp["precio+fund"] - comp["precio"]
print(comp.round(4)) precio precio+fund difference
2020 0.0109 0.0175 0.0066
2021 0.0204 0.0182 -0.0022
2022 0.0121 0.0171 0.0051
2023 -0.0023 0.0013 0.0036
2024 0.0492 0.0515 0.0022
2025 0.0372 0.0402 0.0030Adding valor_bp and sue improves the IC in five of the six years, with differences from 0.002 to 0.007, and makes
it slightly worse in 2021. No yearly difference is large compared with the noise of a weekly IC, so the honest
conclusion is the mean's: a small and fairly consistent improvement, which counts as one more trial.
- Write a gate (
puede_registrarse(run)) that blocks registering a run if it lacks thegit_commitordatos_md5tag, if its status is notFINISHED, if it uses any feature from a blocklist or if its mean IC exceeds the ceiling. Run it over the seven runs.
Solution
NEGRA = {"revision_analistas"}
def puede_registrarse(r):
motivos = []
for t in ("git_commit", "datos_md5"):
if not isinstance(r.get(f"tags.{t}"), str):
motivos.append(f"missing {t}")
if r["status"] != "FINISHED":
motivos.append("did not finish")
usadas = set(str(r["params.features"]).split(","))
if usadas & NEGRA:
motivos.append(f"uses {sorted(usadas & NEGRA)}")
if r["metrics.ic_medio"] > TECHO:
motivos.append("IC above the ceiling")
return motivos
for _, r in runs.iterrows():
print(f"{r['tags.mlflow.runName']:25}",
puede_registrarse(r) or "ok")precio+fund+rev a=1 ["uses ['revision_analistas']", 'IC above the ceiling']
precio+fund+rev a=100000 ["uses ['revision_analistas']", 'IC above the ceiling']
precio+fund a=1 ok
precio+fund a=100000 ok
precio a=100000 ok
precio a=1 ok
precio+fund+sent a=1 ['did not finish']The two leaky runs are rejected for two reasons (the blocklisted feature and the impossible IC), the failed one
because it did not finish, and the four honest ones pass. In a real project, the gate goes in the same code that
registers, right before register_model, so nobody skips it by forgetting.
- A colleague proposes storing the data in git with Git LFS and the experiments in a spreadsheet. What would the team lose compared with the lesson's combination?
Solution
Git LFS also stores pointers in git and the content elsewhere, so for versioning large files it is a reasonable
alternative; what it does not give you is reading a version from Python by commit (dvc.api.open), pipelines, or a
choice of remotes other than the LFS server. The spreadsheet is the serious problem: nobody records failures in it, or
the data md5, or the per-year metric, or the commit, and it is edited by hand without a trace. The lesson's log writes
itself, from the same code that trains, including the runs that fail; that is what makes it possible to count the
trials for the Deflated Sharpe (lesson 2.2.4) and to reproduce a result months later. And the model registry links
each deployed version to the run that produced it, which a spreadsheet does not.
Footnotes
-
DVC, “Get Started: Data Versioning” (https://dvc.org/doc/start/data-management/data-versioning):
dvc addcreates the.dvcfile with the md5, the cache goes in.dvc/cache/files/md5,dvc push/dvc pullwork with the remote, and to go back to a version,git checkoutof the.dvcfile anddvc checkout. Accessed on October 23, 2026. ↩ -
MLflow, “Model Registry” (https://mlflow.org/docs/latest/ml/model-registry/): unique name, versions starting at 1, tags, the
models:/<name>/<version>URI and aliases likemodels:/MyModel@champion. “Tracking” (https://mlflow.org/docs/latest/ml/tracking/) describes the stores andmlflow server. Accessed on October 23, 2026. The default SQLite store and the file store warning are those of mlflow-skinny 3.17.2, checked in the course environment. ↩ -
DVC, “dvc.api.open()” (https://dvc.org/doc/api-reference/open): opens a file versioned by DVC in a DVC project;
revis the git revision (commit, branch or tag). Accessed on October 23, 2026. ↩ -
Weights & Biases, “Experiments overview” (https://docs.wandb.ai/guides/track/):
wandb.initas a context manager,run.config,run.logandrun.log_artifact; the page is served as “CoreWeave Docs”. Deployment options, including Self-Managed: https://docs.wandb.ai/platform/hosting/. Accessed on October 23, 2026. ↩
Each answer is folded under its question
Self-assessment
After
dvc add datos/panel_semanal.parquetandgit commit, what does the git repository contain?- AThe whole parquet file, compressed by git
- BNothing: DVC replaces git
- CA copy of the parquet file in
.git/lfs - DThe
panel_semanal.parquet.dvcfile with the md5 and the size, and a.gitignorethat excludes the parquet file; the data goes to the DVC cache and to the remote
Show answer
Answer D. The
panel_semanal.parquet.dvcfile with the md5 and the size, and a.gitignorethat excludes the parquet file; the data goes to the DVC cache and to the remoteDVC leaves a small, readable pointer in git. The content lives in
.dvc/cache(addressed by its md5) and, afterdvc push, on the remote. That is why a commit identifies an exact version of the data without putting megabytes into the history.One of the logged runs gives a mean IC of 0.176 with a t of 58. What does the tracking log say?
- AThat it is the best model and should be registered
- BThat its parameters include
revision_analistas, the leaky feature from level 5, and that its IC exceeds the oracle's ceiling (0.029): it is impossible out of sample - CThat the alpha was badly chosen
- DNothing: MLflow does not store the features
Show answer
Answer B. That its parameters include
revision_analistas, the leaky feature from level 5, and that its IC exceeds the oracle's ceiling (0.029): it is impossible out of sampleThe log stores which features each run used. A result better than the theoretical ceiling is leakage, not a discovery, and the feature list says which one. Without the log, that number could end up in a report with nobody knowing where it came from.
The run with
sentimientofails because the feature has empty values. Why is it worth keeping it logged asFAILED?- ABecause it counts as a trial: the number of attempts is what the Deflated Sharpe needs, and failures are also decisions made by looking at the data
- BBecause MLflow does not allow deleting it
- CBecause it takes up little space
- DIt is not worth it: failures should be deleted so the report stays clean
Show answer
Answer A. Because it counts as a trial: the number of attempts is what the Deflated Sharpe needs, and failures are also decisions made by looking at the data
Every variant tried, whether it works or not, is a look at the data (lesson 2.2.4). If failures disappear, the number of trials is underestimated and the multiple-testing correction falls short.
The vendor corrects the data and
dvc addproduces a new md5. How do you recover the data the chosen run was trained on?- AYou cannot: DVC only keeps the latest version
- BBy downloading the data from the vendor again
- CWith the commit stored in the run's
git_committag:dvc.api.open(…, rev=commit)reads that version (orgit checkoutof the.dvcfile anddvc checkout), and its md5 matches thedatos_md5tag - DWith
mlflow.load_model
Show answer
Answer C. With the commit stored in the run's
git_committag:dvc.api.open(…, rev=commit)reads that version (orgit checkoutof the.dvcfile anddvc checkout), and its md5 matches thedatos_md5tagThe DVC cache and remote keep each version by its md5, and the
.dvcfile of each commit says which one belongs to it. The run'sdatos_md5tag lets you check that what you recovered is the same, byte for byte.What is the
candidatoalias of the registered model for?- ATo encrypt the model
- BSo the code that uses it asks for
models:/ridge-semanal@candidatoand not a version number: changing versions means moving the alias, without touching that code - CTo delete old versions
- DSo MLflow retrains the model
Show answer
Answer B. So the code that uses it asks for
models:/ridge-semanal@candidatoand not a version number: changing versions means moving the alias, without touching that codeAn alias is a named reference to a specific version, which can be reassigned. Lesson 7.3.2 uses it for the champion/challenger scheme.
Key points
- Reproducing a number takes five things: the exact data, the exact code, the parameters, the environment and the seeds. Git stores the code, DVC the data (by md5) and MLflow ties the two together in each run.
- DVC leaves a
.dvcfile in git with the md5 and the size; the content goes to the cache and to a remote. A commit identifies an exact version of the data, anddvc.api.open(…, rev=…)reads it. - Each MLflow run stores parameters, tags (commit, data md5), per-step metrics and status; the ones that fail also count as trials for the Deflated Sharpe.
- The log exposes the leakage: the run with
revision_analistasgives an IC of 0.176, six times the oracle's ceiling (0.029); the honest one, 0.024. - The model registry provides versions, tags and aliases: the code asks for
models:/ridge-semanal@candidatoand gets the same predictions as the scikit-learn model. - After the vendor correction, the data of the chosen run is recovered from its commit, its md5 matches the tag and the IC comes out identical; with the new data, it does not.