Skip to main content

Experiment Tracking

Experiment tracking answers “what happened in this execution?” A useful run record connects intent, immutable inputs, parameters, environment, metrics, and output artifacts. A dashboard is optional; the evidence contract is not.

Prerequisites​

Complete Data and Code Versioning. Be able to explain why a metric without its evaluation population is ambiguous.

Learning goals​

  • define a run independently of a tracking product.
  • separate parameters, metrics, artifacts, and lineage.
  • compare Study Sprint runs without selecting on the holdout opportunistically.

The conceptual record​

A run is one execution of a declared recipe. It should answer:

  • Why was the run made, and who or what initiated it?
  • Which code, configuration, data, and environment were used?
  • Which parameters were effective?
  • Which metrics were calculated on which named split and metric definition?
  • Which artifacts were produced, and how are their bytes identified?
  • Did validation pass, fail, or remain incomplete?

MLflow and other tools implement variants of runs, parameters, metrics, tags, and artifacts. Learn that model first; then choose a tool because its storage, access control, retention, comparison, and lineage behavior fit the organization.

Worked example​

The local Study Sprint run.json is a deliberately small tracking backend:

{
  "artifacts": {
    "metrics.json": {"sha256": "<digest>"},
    "model.json": {"sha256": "<digest>"},
    "predictions.csv": {"sha256": "<digest>"}
  },
  "code_version": "ticket-39-fixture-v1",
  "parameters": {
    "random_state": 39,
    "test_ids": ["s04", "s09", "s14", "s19", "s24", "s29", "s34", "s39"]
  },
  "schema_version": 1
}

metrics.json reports accuracy, log loss, ROC AUC, and row counts. predictions.csv preserves entity-level holdout outputs so a reviewer can recompute metrics and inspect the one classification error. That file can contain sensitive information in real systems; apply access control and retention rather than logging raw predictions by default.

Tracking is not validation​

A tracking server can faithfully store a leaked metric, mislabeled dataset, or cherry-picked run. Decide comparison rules before looking at the holdout. Record metric definitions, positive class, threshold, evaluation window, slice definitions, and uncertainty where relevant. Mark exploratory runs rather than promoting the winner of many undocumented attempts.

Do not make timestamps the primary run identity: retries can occur at the same logical time, clocks can differ, and timestamps do not bind inputs. A content-derived or generated run ID may help, but identical declared inputs do not necessarily imply identical hardware execution. Preserve both identity and execution metadata.

Common mistakes​

  • Logging only the best score and deleting failed runs.
  • Comparing metrics computed on different splits or label definitions.
  • Recording parameter files without resolved overrides.
  • Uploading secrets, personal data, or unrestricted predictions as artifacts.
  • Assuming a tracking UI provides a model registry, approval workflow, or deployment control automatically.

Practice​

Design a comparison table for two Study Sprint candidates. Include the minimum columns needed to determine whether the metrics are comparable. Add one field that prevents a reviewer from mistaking exploration for an approved run.

Next steps​

Testing Data and ML Code converts important run assumptions and outcomes into executable checks.

Primary sources​

Checked 2026-10-15: MLflow Tracking concepts, W3C PROV data model, and NIST AI Risk Management Framework.