Skip to main content

DVC Data Pipelines

A pipeline tool should make causality between files visible: which command read which dependency, which parameters mattered, and which outputs became stale. It cannot decide whether a dataset is lawful, representative, correctly labeled, or safe to retain. The Study Sprint command already owns those semantic checks.

Prerequisites​

Complete MLflow Tracking Locally and Reproducible Training Pipelines.

Learning goals​

  • describe stages through commands, dependencies, parameters, outputs, and metrics.
  • distinguish content-addressed reproducibility from data meaning and governance.
  • choose DVC, a build tool, or a plain script from actual constraints.

Worked example​

DVC 3.67.1 is the stable PyPI release checked for this lesson. A small dvc.yaml could expose the Study Sprint training stage:

stages:
  train:
    cmd: >-
      python examples/ml-engineering/train.py
      --config examples/ml-engineering/training-config.json
      --output-dir build/study-sprint
      --code-version reviewed-commit
    deps:
      - examples/ml-engineering/train.py
      - examples/ml-engineering/training-config.json
      - fixtures/ml-engineering/study-sprint-model.csv
    outs:
      - build/study-sprint/model.json
      - build/study-sprint/predictions.csv
      - build/study-sprint/run.json
    metrics:
      - build/study-sprint/metrics.json:
          cache: false

After dvc repro, DVC records resolved dependency and output hashes in dvc.lock; dvc metrics show reads the declared metric. Review both dvc.yaml and the lockfile. A cache saves bytes and recomputation but is not a backup, legal basis, schema contract, or catalog.

This is a manual boundary: the repository does not install DVC because the executable capstone needs no remote cache or DVC runtime. Test the snippet only in a separate environment pinned to dvc==3.67.1. Adding DVC merely to run one deterministic command would increase dependency and cache ownership without strengthening the public seam.

A plain script or make-style target is often sufficient when inputs are small, committed, and the graph has few stages. DVC becomes useful for large uncommitted artifacts, content-addressed reuse, parameter experiments, or teams that already own cache access, retention, and recovery.

Common mistakes​

  • Omitting code or configuration from deps, then reusing a stale output.
  • Declaring a directory so broadly that unrelated edits invalidate every stage.
  • Treating a hash match as proof of semantic quality or consent.
  • Depending on an unavailable remote cache without a recovery plan.
  • Mixing promotion or deployment approval into automatic reproduction.

Practice​

Add hypothetical serve-check and monitor stages after train. Identify their precise dependencies and explain which outputs should be cached, committed, or rebuilt.

Next steps​

Continue with Feature Stores and Feast to examine time-aware feature retrieval and online consistency.

Primary sources​

Checked 2026-10-15: DVC defining pipelines, DVC metrics and parameters, DVC lockfile, and PyPI DVC 3.67.1.