Lesson 7.3.1 · 70 min

Tracking experiments, models and data: MLflow, DVC and a run reproduced end to end

A number in a research report is worth as much as the ability to reproduce it. This lesson puts the level 5 weekly panel under DVC and the code under git, logs seven runs of a ridge model in MLflow (including one that fails and one with leakage), registers the chosen model with its data version and, after a vendor correction, recovers the exact data of that run and gets the same IC.

Fig. — Mean weekly out-of-sample IC (2020-2025) of the seven logged runs of the cross-sectional ridge: the two with revision_analistas exceed the oracle ceiling (0.029) six times over; the chosen one gives 0.024 and one fails
Level 7 lessons
  1. 7.1 · Software engineering for trading

    1. 7.1.1 Project structure, configuration and quality: from notebook to repository
    2. 7.1.2 Performance: measure, profile and compile only what is needed
    3. 7.1.3 Patterns: events, message queues and hexagonal architecture
  2. 7.2 · Infrastructure and deployment

    1. 7.2.1 Docker and docker-compose: reproducible images with uv
    2. 7.2.2 VPS, cloud and latency; CI/CD with GitHub Actions
    3. 7.2.3 Orchestration and monitoring: task graphs, structured logs, metrics, alerts and heartbeats
    4. 7.2.4 Security and continuity: 2FA, IP allowlists, backups and failure procedures
  3. 7.3 · MLOps for trading models

    1. 7.3.1 Tracking experiments, models and data: MLflow, DVC and a run reproduced end to end
    2. 7.3.2 Scheduled retraining, automatic validation and champion/challenger
    3. 7.3.3 Live model monitoring and automatic shutdown
    4. 7.3.4 Model serving: ONNX, CPU or GPU inference and the cost of LLMs in production
  4. 7.4 · Regulation, ethics and taxation

    1. 7.4.1 MiFID II: clients, suitability and algorithmic trading
    2. 7.4.2 ESMA and CFDs: leverage, close-out and negative balance
    3. 7.4.3 PRIIPs, the KID and US ETFs
    4. 7.4.4 MiCA after the transitional period: checking a provider in the registers
    5. 7.4.5 Market abuse: spoofing, layering and the bot that imitates them by accident
    6. 7.4.6 Trading taxes in Spain
    7. 7.4.7 Ethics: personal data, artificial intelligence and selling signals
  5. 7.5 · Quant career

    1. 7.5.1 Quant roles: six jobs around the same system
    2. 7.5.2 Where to work: what each firm does and what the registers say
    3. 7.5.3 Public portfolio and interviews: show judgment, not returns
    4. 7.5.4 Certifications, training and prop firms: what each one certifies and what a challenge measures
  6. 7.6 · Final capstone

    1. 7.6.1 Capstone: common requirements, the panel's rubric and preparing the defense
    2. 7.6.2 Capstone C1: a multi-asset futures system, from forecast to paper account
    3. 7.6.3 Capstone C2: systematic options book, volatility surface and real-time Greeks
    4. 7.6.4 Capstone C3: stock and ETF portfolio with ML, HRP and Spanish taxation
    5. 7.6.5 Capstone C4: a research platform with LLMs and alternative data
    6. 7.6.6 Capstone C5: crypto market making or arbitrage with counterparty control

What you need to be able to reproduce

Lesson 2.2.4 built an experiment log in a JSONL file, with the fingerprint of the data and of the code on each line and a hash chain to detect edits. Lesson 5.4.4 logged twenty variants of a model so the Deflated Sharpe could be computed at the end, and its last exercise listed the fields a real log would need. This lesson stores them with the tools used in production, and checks at the end that they are enough to reproduce a result.

Reproducing a number takes five things: the exact data, the exact code, the parameters, the environment (library versions) and the seeds. The environment is pinned by uv.lock (lesson 0.2.2) and the Docker image (lesson 7.2.1); seeds are parameters. That leaves three pieces and three tools: git for the code, DVC for the data, which does not fit in git, and MLflow for what ties the two together in each run.

Code: gitRunData: DVCParametersMetricsModelRegistry
Mermaid source
flowchart TB
  G[Code: git] --> E[Run]
  D[Data: DVC] --> E
  P[Parameters] --> E
  E --> M[Metrics]
  E --> O[Model]
  O --> R[Registry]

The experiment is the cross-sectional ridge from level 5: predict next week's return of 400 synthetic stocks from their weekly features, training up to 2019 and measuring the IC (the weekly rank correlation between prediction and outcome) from 2020 to 2025. The level 5 data has two known traps: revision_analistas is a leak, and sentimiento is empty for stocks without coverage. None of the runs in this lesson is a good model; what matters is what the log says about each one.

The dataset and the code

First, the experiment's dataset: eight features converted to cross-sectional ranks and the label in excess of each week's mean, in a parquet file inside the folder of a new project.

IN [01]python
import hashlib
from pathlib import Path

import numpy as np
import pandas as pd

COLS = ["ret_1s", "mom_12_1", "vol_63d", "volumen_rel", "valor_bp",
        "sue", "sentimiento", "revision_analistas"]
f = pd.read_parquet("panel5_features.parquet",
                    columns=["fecha", "ticker"] + COLS)
e = pd.read_parquet("panel5_etiquetas.parquet",
                    columns=["fecha", "ticker", "ret_sig_semana"])
df = f.merge(e, on=["fecha", "ticker"]).dropna(
    subset=["ret_sig_semana"])
for c in COLS:                     # cross-sectional rank in [-0.5, 0.5]
    df[c] = df.groupby("fecha")[c].rank(pct=True) - 0.5
df["y"] = (df["ret_sig_semana"]
           - df.groupby("fecha")["ret_sig_semana"].transform("mean"))
panel = df[["fecha", "ticker"] + COLS + ["y"]].astype(
    {c: "float32" for c in COLS + ["y"]})

RUTA = Path("proyecto/datos/panel_semanal.parquet")
RUTA.parent.mkdir(parents=True)
panel.to_parquet(RUTA, index=False)

def md5(ruta):
    return hashlib.md5(Path(ruta).read_bytes()).hexdigest()

print(panel.shape, panel["fecha"].min().date(),
      panel["fecha"].max().date())
print(f"{RUTA.stat().st_size / 1e6:.1f} MB · md5 {md5(RUTA)}")
print(panel["sentimiento"].isna().mean().round(3),
      "empty in sentimiento")
OUT [01]The experiment's weekly panel
(291600, 11) 2012-01-13 2025-12-26
4.5 MB · md5 36eccd106ec9200919d2c447fffccac2
0.32 empty in sentimiento

That is 291,600 rows (400 stocks over 729 weeks with a label), from January 2012 to December 2025, in 4.5 MB. 32% of the sentimiento values are empty: the rank of a stock with no data point is also empty. The md5 is the fingerprint of the content, and changing a single value changes it completely. Lesson 2.2.4 computed it by hand; from here on DVC computes and stores it.

The training code goes in a project file, not in the cell: what git versions is the file, and the commit identifies exactly the code that produced each number. The weekly IC is the Pearson correlation between the ranks of the prediction and of the label within each week, computed with group aggregates:

IN [02]python
import sys

CODIGO = '''\
"""Ridge transversal y su IC semanal fuera de muestra."""
from sklearn.linear_model import Ridge

CORTE = "2020-01-01"


def ic_semanal(datos, pred):
    r = datos[["fecha", "y"]].assign(p=pred)
    r[["p", "y"]] = r.groupby("fecha")[["p", "y"]].rank()
    g = r.groupby("fecha")
    cov = (r["p"] * r["y"]).groupby(r["fecha"]).mean() - (
        g["p"].mean() * g["y"].mean())
    return cov / (g["p"].std(ddof=0) * g["y"].std(ddof=0))


def entrenar(panel, features, alpha):
    tr = panel[panel["fecha"] < CORTE]
    te = panel[panel["fecha"] >= CORTE]
    modelo = Ridge(alpha=alpha).fit(tr[features], tr["y"])
    return modelo, ic_semanal(te, modelo.predict(te[features]))
'''
Path("proyecto/entrenar.py").write_text(CODIGO)
Path("proyecto/.gitignore").write_text("__pycache__/\n")
sys.path.insert(0, "proyecto")
import entrenar

The file's docstring (“cross-sectional ridge and its out-of-sample weekly IC”) and the commit messages further down stay as in the Spanish edition of the course: they are part of what git hashes, and changing them would change the commit identifiers the lesson quotes.

Versioning the data with DVC

A parquet file of several megabytes should not go into git: every version would stay forever in the history of every clone. DVC solves this with a pointer: dvc add computes the file's md5, copies it to a local cache (.dvc/cache, addressed by that md5), writes a small .dvc file with the md5 and the size, and adds the original to .gitignore. Git versions the .dvc file; dvc push uploads the content to a remote (here, a local folder; in production, S3, Azure, GCS, SSH…) and dvc pull downloads it1Note 1 · DVC, “Get Started: Data Versioning” (https://dvc.org/doc/start/data-management/data-versioning): dvc add creates the .dvc file with the md5, the cache goes in .dvc/cache/files/md5, dvc push/dvc pull work with the remote, and to go back to a version, git checkout of the .dvc file and dvc checkout. Accessed on October 23, 2026..

panel.parquetDVC cacheRemotepanel.parquet.dvcGit repository
Mermaid source
flowchart TB
  W[panel.parquet] --> C[DVC cache]
  C --> R[Remote]
  W --> P[panel.parquet.dvc]
  P --> G[Git repository]
IN [03]python
import subprocess

def sh(*cmd):
    r = subprocess.run(cmd, cwd="proyecto", capture_output=True,
                       text=True, check=True)
    return (r.stdout + r.stderr).strip()

sh("git", "init", "-q")
sh("dvc", "init", "-q")
sh("dvc", "add", "-q", "datos/panel_semanal.parquet")
print(sh("dvc", "remote", "add", "-d", "local", "../remoto-dvc"))
sh("git", "add", "-A")
sh("git", "commit", "-q", "-m", "Panel semanal v1 y entrenar.py")
sh("dvc", "push", "-q")
COMMIT_V1 = sh("git", "rev-parse", "--short", "HEAD")

print("commit", COMMIT_V1)
print(Path("proyecto/datos/panel_semanal.parquet.dvc").read_text())
print(sh("git", "ls-files"))
remoto = [p for p in Path("remoto-dvc").rglob("*") if p.is_file()]
print(len(remoto), "file on the remote:",
      remoto[0].relative_to("remoto-dvc").parts[:2])
OUT [03]Data and code under control: git and DVC
Setting 'local' as a default remote.
commit e77070e
outs:
- md5: 36eccd106ec9200919d2c447fffccac2
  size: 4529279
  hash: md5
  path: panel_semanal.parquet

.dvc/.gitignore
.dvc/config
.dvcignore
.gitignore
datos/.gitignore
datos/panel_semanal.parquet.dvc
entrenar.py
1 file on the remote: ('files', 'md5')

Only dvc remote add -d prints anything. The .dvc file is what goes into git: the md5, the size in bytes (4,529,279), the algorithm and the path. git ls-files confirms the parquet file is not in the repository: there are the .dvc file, the code, the DVC configuration and a datos/.gitignore that dvc add created to exclude the parquet file. The remote has one file, stored under files/md5/ with its md5 as the name. Commit e77070e identifies both the code and the data version; it comes out the same in every run of the lesson because the content, the author and the date of the commit are fixed.

An experiment in MLflow

MLflow organizes work into experiments that contain runs. Each run stores parameters (what was chosen), metrics (what was measured, optionally per step), tags (free context), artifacts (files, models) and its status. Without a configured server, the MLflow 3.17 client in the environment creates an SQLite database (mlflow.db) in the working folder and stores artifacts in mlruns/; the general documentation still describes the mlruns file store as the default, but that version no longer uses it without an explicit variable. The grid: three feature sets, two values of alpha and one more variant with sentimiento. Each run carries the code commit and the data md5 as tags:

IN [04]python
import os
import mlflow

mlflow.set_experiment("ridge-semanal")
DATOS_MD5 = md5(RUTA)
BASE = ["ret_1s", "mom_12_1", "vol_63d", "volumen_rel"]
REJILLA = [("precio", BASE, a) for a in (1.0, 1e5)]
REJILLA += [("precio+fund", BASE + ["valor_bp", "sue"], a)
            for a in (1.0, 1e5)]
REJILLA += [("precio+fund+rev", BASE + ["valor_bp", "sue",
             "revision_analistas"], a) for a in (1.0, 1e5)]
REJILLA += [("precio+fund+sent", BASE + ["valor_bp", "sue",
             "sentimiento"], 1.0)]

for conjunto, feats, alpha in REJILLA:
    try:
        with mlflow.start_run(run_name=f"{conjunto} a={alpha:g}"):
            mlflow.set_tags({"git_commit": COMMIT_V1,
                             "datos_md5": DATOS_MD5,
                             "datos": "datos/panel_semanal.parquet"})
            mlflow.log_params({"conjunto": conjunto, "alpha": alpha,
                               "features": ",".join(feats),
                               "corte": entrenar.CORTE})
            modelo, ic = entrenar.entrenar(panel, feats, alpha)
            for anio, v in ic.groupby(ic.index.year).mean().items():
                mlflow.log_metric("ic_anual", v, step=anio)
            mlflow.log_metrics({
                "ic_medio": ic.mean(),
                "ic_t": ic.mean() / ic.std() * np.sqrt(len(ic))})
    except ValueError as err:
        print(f"{conjunto} a={alpha:g}:", str(err).splitlines()[0])

print([p for p in ("mlflow.db", "mlruns") if os.path.exists(p)])
OUT [04]Seven logged runs
precio+fund+sent a=1: Input X contains NaN.
['mlflow.db']

Six runs finish and one fails: scikit-learn's ridge does not accept empty values, and sentimiento has them. The exception leaves the with block, and MLflow closes the run with status FAILED before the except catches it: the attempt stays logged with its parameters and tags. In the working folder mlflow.db appears and mlruns does not yet, because no run has stored artifacts.

Search, compare and count

Runs are queried with mlflow.search_runs, which returns a DataFrame with one column per parameter, metric and tag, and accepts filters with a syntax similar to SQL. The DataFrame also carries identifiers and times, which change on every run: you select the columns you care about. The feature-set names (precio, precio+fund…) are the run identifiers and stay as they are.

IN [05]python
runs = mlflow.search_runs(experiment_names=["ridge-semanal"],
                          order_by=["metrics.ic_medio DESC"])
vista = runs[["tags.mlflow.runName", "status", "metrics.ic_medio",
              "metrics.ic_t"]].set_axis(
    ["run", "status", "ic_mean", "t"], axis=1)
print(vista.round({"ic_mean": 4, "t": 1}).to_string(index=False))
print("logged trials:", len(runs), "· failed:",
      int((runs["status"] == "FAILED").sum()))

TECHO = 0.029                      # oracle IC (level 5 README)
sosp = mlflow.search_runs(experiment_names=["ridge-semanal"],
                          filter_string=f"metrics.ic_medio > {TECHO}")
extra = {f for fs in sosp["params.features"]
         for f in fs.split(",")} - set(BASE + ["valor_bp", "sue"])
print("above the ceiling:", len(sosp), "· extra features:", extra)

cli = mlflow.MlflowClient()
r0 = runs[runs["tags.mlflow.runName"] == "precio+fund a=1"].iloc[0]
hist = cli.get_metric_history(r0["run_id"], "ic_anual")
print({m.step: round(m.value, 3) for m in hist})
OUT [05]The runs, sorted by IC
                     run   status  ic_mean    t
     precio+fund+rev a=1 FINISHED   0.1763 58.7
precio+fund+rev a=100000 FINISHED   0.1763 58.6
         precio+fund a=1 FINISHED   0.0243  4.5
    precio+fund a=100000 FINISHED   0.0243  4.5
         precio a=100000 FINISHED   0.0213  3.7
              precio a=1 FINISHED   0.0212  3.6
    precio+fund+sent a=1   FAILED      NaN  NaN
logged trials: 7 · failed: 1
above the ceiling: 2 · extra features: {'revision_analistas'}
{2020: 0.018, 2021: 0.018, 2022: 0.017, 2023: 0.001, 2024: 0.051, 2025: 0.04}

The first two runs give an IC of 0.176 with a t of 58: six times the IC of the data's oracle (0.029), the maximum achievable out of sample. The metrics.ic_medio > 0.029 filter finds them, and the difference between their features and those of the others points to revision_analistas, the leaky feature from level 5. Without the features parameter in the log, that number would have been read as a discovery. The honest ones give 0.021 with the price features and 0.024 adding valor_bp and sue, with t from 3.6 to 4.5; alpha changes almost nothing with more than 160,000 training rows. Seven trials were logged, one of them failed: that 7 is the number of attempts the Deflated Sharpe needs (lessons 2.2.4 and 5.4.4), and it would come out lower if failures were not logged. The figure at the top shows all seven together. Finally, the yearly history of the chosen run shows that the IC is not stable: almost zero in 2023 and 0.05 in 2024.

What to log in a trading experiment

The lesson's grid logs the minimum. A trading experiment that will end up in a report or in production needs more fields, and each one has a natural place:

WhatWhereExample
Data versiontagdatos_md5, the commit with the .dvc file
Code versiontaggit_commit; no uncommitted changes when running
Validationparameterscutoff, window, purging and embargo (lessons 5.4.2 and 5.4.3)
Costsparameterscost model and execution assumptions
Seedsparameterssemilla of the model and of the sampling
Summary resultmetricsmean IC and t, net Sharpe, turnover
Result over timeper-step metricsIC per year, drawdown per year
Full seriesartifactweekly returns or IC, for the Deflated Sharpe and the PBO
Decisiontag“discarded: leakage in revision_analistas”

What does not go in: credentials (parameters and tags are readable in plain text, lesson 4.4.5), personal data and whole datasets as artifacts (that is what DVC is for). And a process rule: the log is written from the same code that trains, not by hand afterwards, because what gets written down by hand is what worked.

From the experiment to the model registry

A run stores what was tried; the model registry stores what will be used. A registered model has a unique name and numbered versions, each linked to the run that produced it; versions accept tags and aliases, named references that can be moved from one version to another, like models:/MyModel@champion in the MLflow documentation2Note 2 · MLflow, “Model Registry” (https://mlflow.org/docs/latest/ml/model-registry/): unique name, versions starting at 1, tags, the models:/<name>/<version> URI and aliases like models:/MyModel@champion. “Tracking” (https://mlflow.org/docs/latest/ml/tracking/) describes the stores and mlflow server. Accessed on October 23, 2026. The default SQLite store and the file store warning are those of mlflow-skinny 3.17.2, checked in the course environment.. The chosen model is saved inside its run with a signature (the input columns and types, inferred from the example) and registered:

IN [06]python
feats = r0["params.features"].split(",")
modelo, ic = entrenar.entrenar(panel, feats, float(r0["params.alpha"]))
X_ej = panel.loc[panel["fecha"] == panel["fecha"].max(), feats].head(3)

with mlflow.start_run(run_id=r0["run_id"]):
    info = mlflow.sklearn.log_model(
        modelo, name="modelo", input_example=X_ej,
        registered_model_name="ridge-semanal")
version = info.registered_model_version
cli.set_model_version_tag("ridge-semanal", version, "datos_md5",
                          DATOS_MD5)
cli.set_registered_model_alias("ridge-semanal", "candidato", version)

mv = cli.get_model_version_by_alias("ridge-semanal", "candidato")
print("version", mv.version, "· from the chosen run:",
      mv.run_id == r0["run_id"], "· tags:", mv.tags)
pf = mlflow.pyfunc.load_model("models:/ridge-semanal@candidato")
print(pf.metadata.get_input_schema())
print(np.allclose(pf.predict(X_ej), modelo.predict(X_ej)))
OUT [06]Register the chosen model and load it by its alias
Successfully registered model 'ridge-semanal'.
Created version '1' of model 'ridge-semanal'.
version 1 · from the chosen run: True · tags: {'datos_md5': '36eccd106ec9200919d2c447fffccac2'}
['ret_1s': float (required), 'mom_12_1': float (required), 'vol_63d': float (required), 'volumen_rel': float (required), 'valor_bp': float (required), 'sue': float (required)]
True

MLflow reports that it created the registered model and its version 1. The candidato (candidate) alias points to that version, which links to the chosen run and carries the tag with the data md5. The signature, inferred from the input example, is the six columns with their type: a predict call missing any of them raises an exception instead of returning meaningless numbers. And the model loaded by its alias through the generic interface (pyfunc), as a service that knows nothing about scikit-learn would do, gives the same predictions as the original. The code that uses it asks for models:/ridge-semanal@candidato; when there is a better version, the alias moves and that code does not change.

A vendor correction

Data changes. The vendor announces that the 2023 SUE values were published in the file one week early: each value was actually known one week later. The corrected version (v2) shifts sue by one week within 2023, and the workflow is the same as with the code: modify, dvc add, commit and dvc push.

IN [07]python
v2 = panel.copy()
en_2023 = v2["fecha"].dt.year == 2023
v2.loc[en_2023, "sue"] = (v2[en_2023].groupby("ticker")["sue"]
                          .shift(1).fillna(0))
v2.to_parquet(RUTA, index=False)
print(sh("dvc", "status"))
sh("dvc", "add", "-q", "datos/panel_semanal.parquet")
sh("git", "commit", "-q", "-am", "Panel semanal v2: SUE de 2023")
sh("dvc", "push", "-q")
print(sh("git", "log", "--oneline"))
print("md5 v1", DATOS_MD5[:12], "· md5 v2", md5(RUTA)[:12])
print(sh("dvc", "diff", "HEAD~1"))
OUT [07]Data v2: correct, version and push
datos/panel_semanal.parquet.dvc:
	changed outs:
		modified:           datos/panel_semanal.parquet
3562fe4 Panel semanal v2: SUE de 2023
e77070e Panel semanal v1 y entrenar.py
md5 v1 36eccd106ec9 · md5 v2 968bf4c6f9f1
Modified:
    datos/panel_semanal.parquet

files summary: 1 modified

dvc status detects that the file no longer matches its .dvc file. After dvc add and the commit, the history has two versions, each with its md5, and dvc diff HEAD~1 sums up the change: one modified file. Both contents are still in the cache and on the remote; the .dvc file of each commit says which is which.

Reproducing the chosen run

Now the proof that the log is useful. Starting only from the logged run, with nothing from the notebook's memory: its tags say which commit and which md5; dvc.api.open reads the file as it was at that commit, from DVC storage3Note 3 · DVC, “dvc.api.open()” (https://dvc.org/doc/api-reference/open): opens a file versioned by DVC in a DVC project; rev is the git revision (commit, branch or tag). Accessed on October 23, 2026.; its parameters say how to train; and its metric says what should come out.

IN [08]python
import io
import dvc.api

run = mlflow.get_run(r0["run_id"])
tags, params = run.data.tags, run.data.params
with dvc.api.open("datos/panel_semanal.parquet", repo="proyecto",
                  rev=tags["git_commit"], mode="rb") as fh:
    crudo = fh.read()
print("md5 matches the tag:",
      hashlib.md5(crudo).hexdigest() == tags["datos_md5"])

datos_v1 = pd.read_parquet(io.BytesIO(crudo))
f_run = params["features"].split(",")
_, ic_v1 = entrenar.entrenar(datos_v1, f_run, float(params["alpha"]))
_, ic_v2 = entrenar.entrenar(v2, f_run, float(params["alpha"]))
print(f"logged {run.data.metrics['ic_medio']:.6f} · "
      f"reproduced {ic_v1.mean():.6f} · with v2 {ic_v2.mean():.6f}")
print("2023:", round(ic_v1[ic_v1.index.year == 2023].mean(), 4),
      "vs", round(ic_v2[ic_v2.index.year == 2023].mean(), 4))
OUT [08]From the logged run to the same IC
md5 matches the tag: True
logged 0.024276 · reproduced 0.024276 · with v2 0.024311
2023: 0.0013 vs 0.0015

The md5 of what was read matches the tag, and the reproduced IC matches the logged one to the sixth decimal place: 0.024276. With the corrected data, the same code and the same parameters give 0.024311, and the 2023 IC goes from 0.0013 to 0.0015. The difference is small, but enough for a number written with v1 to be unverifiable with v2: without the data version, whoever tries to check that 0.0243 months later will get a different number and will not know why.

In a real project, the code is also recovered from the commit (git worktree add ../rep <commit> or git checkout), and the environment from its uv.lock. Here the code did not change between v1 and v2, so the loaded module is enough. To work with the old version in the project folder, the path in the DVC documentation is to recover that commit's .dvc file and sync:

IN [09]Going back to a commit's data versionpython
import subprocess

subprocess.run(["git", "checkout", "<commit>", "--",
                "datos/panel_semanal.parquet.dvc"], check=True)
subprocess.run(["dvc", "checkout"], check=True)   # or dvc pull

Server, team and alternatives

Everything above works in a folder, which is what a single person needs. With a team, MLflow is served as a tracking server with a shared database (PostgreSQL, MySQL) and an artifact store (S3, Azure, GCS), and the web interface compares runs and plots metrics. The course environment uses mlflow-skinny, the client without server or UI, so these commands are not run:

IN [10]MLflow server and DVC remote in the cloudpython
import subprocess

# tracking server with the lesson's SQLite database
subprocess.run(["mlflow", "server", "--backend-store-uri",
                "sqlite:///mlflow.db", "--port", "5000"])
# in the training code:
# mlflow.set_tracking_uri("http://127.0.0.1:5000")

# DVC remote in a bucket (credentials outside the repository)
subprocess.run(["dvc", "remote", "add", "-d", "nube",
                "s3://mi-bucket/dvc"], cwd="proyecto")

An MLflow server gives access to every experiment and model: it is published on 127.0.0.1 and reached through an SSH tunnel, like Grafana in lesson 7.2.1. The bucket credentials go in the secrets manager or in the process's environment variables (lesson 4.4.5), never in .dvc/config, which goes into git (that is what .dvc/config.local, which is not versioned, is for).

Weights & Biases is the best-known commercial alternative: a cloud service (with a self-hosted installation) with the same model of runs, configuration, per-step metrics and artifacts, plus dashboards to compare them. Its documentation is now published inside CoreWeave's. The basic workflow, according to that documentation4Note 4 · Weights & Biases, “Experiments overview” (https://docs.wandb.ai/guides/track/): wandb.init as a context manager, run.config, run.log and run.log_artifact; the page is served as “CoreWeave Docs”. Deployment options, including Self-Managed: https://docs.wandb.ai/platform/hosting/. Accessed on October 23, 2026.:

IN [11]The same log in Weights & Biasespython
import wandb

config = {"conjunto": "precio+fund", "alpha": 1.0}
with wandb.init(project="ridge-semanal", config=config) as run:
    for anio, v in ic.groupby(ic.index.year).mean().items():
        run.log({"ic_anual": v, "anio": anio})
    run.log({"ic_medio": ic.mean()})
MLflowWeights & BiasesDVC
What it storesruns, metrics, artifacts, registered modelsruns, metrics, artifacts, dashboardsdata and model versions (and pipelines)
Wherefolder, database or your own servercloud servicelocal cache and remote (folder, S3, SSH…)
Licenseopen source (Apache 2.0)commercial serviceopen source (Apache 2.0)
Fitsmodel registry and deployment without depending on third partiesteams that want dashboards without maintaining a serverlarge data that must travel with the code

The choice is secondary. What is not optional is what the cell in the previous section does: from any number in a report you can get to its data, its code and its parameters, and obtain it again. Lesson 7.3.2 starts from this log to retrain, validate and promote models with aliases; lesson 7.3.3 monitors the ones in use, live.

Exercises

  1. Compare the yearly IC history (ic_anual) of the precio a=1 and precio+fund a=1 runs with get_metric_history. In which years does adding valor_bp and sue contribute anything?
Solution
IN [12]python
def historia(nombre):
    rid = runs.loc[runs["tags.mlflow.runName"] == nombre,
                   "run_id"].iloc[0]
    return pd.Series({m.step: m.value for m in
                      cli.get_metric_history(rid, "ic_anual")})

comp = pd.DataFrame({"precio": historia("precio a=1"),
                     "precio+fund": historia("precio+fund a=1")})
comp["difference"] = comp["precio+fund"] - comp["precio"]
print(comp.round(4))
OUT [12]IC per year of two runs
      precio  precio+fund  difference
2020  0.0109       0.0175      0.0066
2021  0.0204       0.0182     -0.0022
2022  0.0121       0.0171      0.0051
2023 -0.0023       0.0013      0.0036
2024  0.0492       0.0515      0.0022
2025  0.0372       0.0402      0.0030

Adding valor_bp and sue improves the IC in five of the six years, with differences from 0.002 to 0.007, and makes it slightly worse in 2021. No yearly difference is large compared with the noise of a weekly IC, so the honest conclusion is the mean's: a small and fairly consistent improvement, which counts as one more trial.

  1. Write a gate (puede_registrarse(run)) that blocks registering a run if it lacks the git_commit or datos_md5 tag, if its status is not FINISHED, if it uses any feature from a blocklist or if its mean IC exceeds the ceiling. Run it over the seven runs.
Solution
IN [13]python
NEGRA = {"revision_analistas"}

def puede_registrarse(r):
    motivos = []
    for t in ("git_commit", "datos_md5"):
        if not isinstance(r.get(f"tags.{t}"), str):
            motivos.append(f"missing {t}")
    if r["status"] != "FINISHED":
        motivos.append("did not finish")
    usadas = set(str(r["params.features"]).split(","))
    if usadas & NEGRA:
        motivos.append(f"uses {sorted(usadas & NEGRA)}")
    if r["metrics.ic_medio"] > TECHO:
        motivos.append("IC above the ceiling")
    return motivos

for _, r in runs.iterrows():
    print(f"{r['tags.mlflow.runName']:25}",
          puede_registrarse(r) or "ok")
OUT [13]A gate before registration
precio+fund+rev a=1       ["uses ['revision_analistas']", 'IC above the ceiling']
precio+fund+rev a=100000  ["uses ['revision_analistas']", 'IC above the ceiling']
precio+fund a=1           ok
precio+fund a=100000      ok
precio a=100000           ok
precio a=1                ok
precio+fund+sent a=1      ['did not finish']

The two leaky runs are rejected for two reasons (the blocklisted feature and the impossible IC), the failed one because it did not finish, and the four honest ones pass. In a real project, the gate goes in the same code that registers, right before register_model, so nobody skips it by forgetting.

  1. A colleague proposes storing the data in git with Git LFS and the experiments in a spreadsheet. What would the team lose compared with the lesson's combination?
Solution

Git LFS also stores pointers in git and the content elsewhere, so for versioning large files it is a reasonable alternative; what it does not give you is reading a version from Python by commit (dvc.api.open), pipelines, or a choice of remotes other than the LFS server. The spreadsheet is the serious problem: nobody records failures in it, or the data md5, or the per-year metric, or the commit, and it is edited by hand without a trace. The lesson's log writes itself, from the same code that trains, including the runs that fail; that is what makes it possible to count the trials for the Deflated Sharpe (lesson 2.2.4) and to reproduce a result months later. And the model registry links each deployed version to the run that produced it, which a spreadsheet does not.

Footnotes

  1. DVC, “Get Started: Data Versioning” (https://dvc.org/doc/start/data-management/data-versioning): dvc add creates the .dvc file with the md5, the cache goes in .dvc/cache/files/md5, dvc push/dvc pull work with the remote, and to go back to a version, git checkout of the .dvc file and dvc checkout. Accessed on October 23, 2026. ↩

  2. MLflow, “Model Registry” (https://mlflow.org/docs/latest/ml/model-registry/): unique name, versions starting at 1, tags, the models:/<name>/<version> URI and aliases like models:/MyModel@champion. “Tracking” (https://mlflow.org/docs/latest/ml/tracking/) describes the stores and mlflow server. Accessed on October 23, 2026. The default SQLite store and the file store warning are those of mlflow-skinny 3.17.2, checked in the course environment. ↩

  3. DVC, “dvc.api.open()” (https://dvc.org/doc/api-reference/open): opens a file versioned by DVC in a DVC project; rev is the git revision (commit, branch or tag). Accessed on October 23, 2026. ↩

  4. Weights & Biases, “Experiments overview” (https://docs.wandb.ai/guides/track/): wandb.init as a context manager, run.config, run.log and run.log_artifact; the page is served as “CoreWeave Docs”. Deployment options, including Self-Managed: https://docs.wandb.ai/platform/hosting/. Accessed on October 23, 2026. ↩

Each answer is folded under its question

Self-assessment

  1. After dvc add datos/panel_semanal.parquet and git commit, what does the git repository contain?

    • AThe whole parquet file, compressed by git
    • BNothing: DVC replaces git
    • CA copy of the parquet file in .git/lfs
    • DThe panel_semanal.parquet.dvc file with the md5 and the size, and a .gitignore that excludes the parquet file; the data goes to the DVC cache and to the remote
    Show answer

    Answer D. The panel_semanal.parquet.dvc file with the md5 and the size, and a .gitignore that excludes the parquet file; the data goes to the DVC cache and to the remote

    DVC leaves a small, readable pointer in git. The content lives in .dvc/cache (addressed by its md5) and, after dvc push, on the remote. That is why a commit identifies an exact version of the data without putting megabytes into the history.

  2. One of the logged runs gives a mean IC of 0.176 with a t of 58. What does the tracking log say?

    • AThat it is the best model and should be registered
    • BThat its parameters include revision_analistas, the leaky feature from level 5, and that its IC exceeds the oracle's ceiling (0.029): it is impossible out of sample
    • CThat the alpha was badly chosen
    • DNothing: MLflow does not store the features
    Show answer

    Answer B. That its parameters include revision_analistas, the leaky feature from level 5, and that its IC exceeds the oracle's ceiling (0.029): it is impossible out of sample

    The log stores which features each run used. A result better than the theoretical ceiling is leakage, not a discovery, and the feature list says which one. Without the log, that number could end up in a report with nobody knowing where it came from.

  3. The run with sentimiento fails because the feature has empty values. Why is it worth keeping it logged as FAILED?

    • ABecause it counts as a trial: the number of attempts is what the Deflated Sharpe needs, and failures are also decisions made by looking at the data
    • BBecause MLflow does not allow deleting it
    • CBecause it takes up little space
    • DIt is not worth it: failures should be deleted so the report stays clean
    Show answer

    Answer A. Because it counts as a trial: the number of attempts is what the Deflated Sharpe needs, and failures are also decisions made by looking at the data

    Every variant tried, whether it works or not, is a look at the data (lesson 2.2.4). If failures disappear, the number of trials is underestimated and the multiple-testing correction falls short.

  4. The vendor corrects the data and dvc add produces a new md5. How do you recover the data the chosen run was trained on?

    • AYou cannot: DVC only keeps the latest version
    • BBy downloading the data from the vendor again
    • CWith the commit stored in the run's git_commit tag: dvc.api.open(…, rev=commit) reads that version (or git checkout of the .dvc file and dvc checkout), and its md5 matches the datos_md5 tag
    • DWith mlflow.load_model
    Show answer

    Answer C. With the commit stored in the run's git_commit tag: dvc.api.open(…, rev=commit) reads that version (or git checkout of the .dvc file and dvc checkout), and its md5 matches the datos_md5 tag

    The DVC cache and remote keep each version by its md5, and the .dvc file of each commit says which one belongs to it. The run's datos_md5 tag lets you check that what you recovered is the same, byte for byte.

  5. What is the candidato alias of the registered model for?

    • ATo encrypt the model
    • BSo the code that uses it asks for models:/ridge-semanal@candidato and not a version number: changing versions means moving the alias, without touching that code
    • CTo delete old versions
    • DSo MLflow retrains the model
    Show answer

    Answer B. So the code that uses it asks for models:/ridge-semanal@candidato and not a version number: changing versions means moving the alias, without touching that code

    An alias is a named reference to a specific version, which can be reassigned. Lesson 7.3.2 uses it for the champion/challenger scheme.

Key points

  • Reproducing a number takes five things: the exact data, the exact code, the parameters, the environment and the seeds. Git stores the code, DVC the data (by md5) and MLflow ties the two together in each run.
  • DVC leaves a .dvc file in git with the md5 and the size; the content goes to the cache and to a remote. A commit identifies an exact version of the data, and dvc.api.open(…, rev=…) reads it.
  • Each MLflow run stores parameters, tags (commit, data md5), per-step metrics and status; the ones that fail also count as trials for the Deflated Sharpe.
  • The log exposes the leakage: the run with revision_analistas gives an IC of 0.176, six times the oracle's ceiling (0.029); the honest one, 0.024.
  • The model registry provides versions, tags and aliases: the code asks for models:/ridge-semanal@candidato and gets the same predictions as the scikit-learn model.
  • After the vendor correction, the data of the chosen run is recovered from its commit, its md5 matches the tag and the IC comes out identical; with the new data, it does not.

Resources

Data