Skip to main content

Reproducible Training Pipelines

A training pipeline is an ordered contract, not merely a scheduler graph. It should make it difficult to fit transformations on holdout data, skip validation, publish half a run, or apply different feature logic at evaluation time.

Prerequisites​

Complete Testing Data and ML Code. Know scikit-learn's Pipeline and ColumnTransformer concepts.

Learning goals​

  • order stages by their data dependencies.
  • keep learned preprocessing inside the fitted estimator pipeline.
  • distinguish idempotent orchestration from deterministic model arithmetic.

The Study Sprint run graph​

The local command performs these stages:

read config + data
        |
validate schema, IDs, values, and classes
        |
partition by declared holdout IDs
        |
fit imputation + scaling + one-hot encoding on train only
        |
fit seeded logistic regression
        |
evaluate the untouched holdout
        |
write model, metrics, predictions, and lineage

The scikit-learn pipeline owns preprocessing and the classifier together. fit sees only training rows; predict_proba applies the learned transformations to the holdout. This prevents the common data leakage pattern of calculating scaling or categories on all rows before the split.

Worked example​

The configured boundary is entity-based rather than row-position-based:

{
  "test_ids": ["s04", "s09", "s14", "s19", "s24", "s29", "s34", "s39"],
  "random_state": 39
}

With the 40-row fixture, the command fits 32 rows and evaluates eight. Reordering the CSV does not change membership. In a time-dependent problem, use an explicit cutoff and preserve event-time/availability rules; a random or entity list split may then be inappropriate.

Repeated execution into fresh directories is safe for this example. A production writer should publish to a temporary location and atomically mark a run complete only after every artifact and checksum succeeds. Retries must use the same logical run identity or deliberately create a linked attempt; silently overwriting an approved artifact destroys lineage.

Determinism has a boundary​

An integer random_state removes one source of variation. Determinism also depends on algorithms, thread scheduling, numeric libraries, hardware, and versions. Some GPU operations are nondeterministic or require slower deterministic modes. Therefore record the envelope, test the outcomes that matter, and use tolerances when bitwise equality is not the requirement.

An orchestrator can make dependencies and retries explicit, but it cannot repair a non-idempotent stage, validate scientific assumptions, or make a stochastic algorithm deterministic by itself.

Common mistakes​

  • Fitting an imputer or encoder before creating the holdout.
  • Allowing category order or feature order to depend on an unordered collection.
  • Publishing metrics before the corresponding model artifact succeeds.
  • Retrying into a mutable “latest” directory without attempt lineage.
  • Assuming a seed controls hardware kernels and all library randomness.

Practice​

Adapt the graph for a temporal churn model. Replace the explicit Study Sprint IDs with a cutoff contract, then state how late-arriving labels, feature availability, and backfills affect repeatability.

Next steps​

Model Artifacts and Safe Serialization examines what to publish and why loading a model is a security boundary.

Primary sources​

Checked 2026-10-15: scikit-learn pipelines and composite estimators, scikit-learn common leakage pitfalls, scikit-learn random_state glossary, and DVC pipeline fundamentals.