Skip to main content

MLflow Tracking Locally

Experiment tracking gives runs a queryable home. It does not make the data correct, the model useful, or an artifact safe to load. The Study Sprint already writes an inert run.json; this lesson shows when a tracking service earns its extra ownership cost.

Prerequisites​

Complete Retraining Policies and Feedback Loops and revisit Experiment Tracking.

Learning goals​

  • map parameters, metrics, tags, inputs, and artifacts to a run identity.
  • configure an explicitly local MLflow backend and artifact directory.
  • decide when a plain file and source control are sufficient.

Worked example​

MLflow 3.17.1 is the stable PyPI release checked for this lesson. Its tracking API still exposes set_tracking_uri, set_experiment, start_run, log_params, log_metrics, and log_artifact. A local adaptation could wrap the existing trainer like this:

from pathlib import Path
import mlflow

root = Path("build/study-sprint").resolve()
mlflow.set_tracking_uri("sqlite:///build/mlflow.db")
mlflow.set_experiment("study-sprint-completion")

with mlflow.start_run(run_name="candidate-1"):
    mlflow.set_tags({"code_version": "reviewed-commit", "model_version": "study-sprint-completion@1"})
    mlflow.log_params({"solver": "liblinear", "random_state": 39})
    mlflow.log_metrics({"accuracy": 0.875, "log_loss": 0.159471, "roc_auc": 1.0})
    for name in ["model.json", "metrics.json", "predictions.csv", "run.json"]:
        mlflow.log_artifact(root / name, artifact_path="training")

This is a manual boundary: MLflow is not installed by the repository or CI, and the snippet does not claim an executed result. To try it, create a separate environment pinned to mlflow==3.17.1, run the existing training command first, and keep the tracking URI local. The capstone executes the same lifecycle evidence with plain JSON files so a hosted server, account, secret, and network are unnecessary.

Choose the run key before execution, log resolved rather than merely requested parameters, retain data/config/code digests, and keep artifacts immutable. SQLite is appropriate for one learner or a small local process; concurrent writers, retention, backup, authentication, and access control require an owned service design.

A plain file is sufficient when runs are few, one repository owns them, diff and grep answer the questions, and no concurrent discovery UI is needed. Introduce MLflow when query volume, collaboration, retention, access policy, or cross-project discovery justifies operating it.

Common mistakes​

  • Logging a mutable path but not its digest or model version.
  • Treating the highest metric in the UI as an approved production model.
  • Storing secrets or sensitive row-level data as parameters, tags, or artifacts.
  • Assuming a local tracking database supplies backup, concurrency, or authorization.
  • Loading an untrusted logged pickle because it came from a tracker.

Practice​

Design a run schema for Study Sprint. Mark which fields belong in parameters, metrics, tags, input lineage, and artifacts. Then explain whether run.json or MLflow is the smaller owned solution for one developer and for twenty teams.

Next steps​

Continue with DVC Data Pipelines to connect file dependencies and reproducible stage execution.

Primary sources​

Checked 2026-10-15: MLflow Tracking, MLflow Tracking API, MLflow tracking-server architecture, and PyPI MLflow 3.17.1.