Skip to main content

Testing Data and ML Code

ML tests must cover more than pure functions. Data is an input with a schema and semantics; a trained model is an observable behavior; the training command is a system boundary. Good tests protect those contracts without freezing private helper functions or exact implementation structure. The Study Sprint fixture makes each layer small enough to inspect.

Prerequisites​

Complete Experiment Tracking. Understand why preprocessing must be fitted only on training data.

Learning goals​

  • distinguish data-contract, training-contract, and model-behavior tests.
  • write assertions against public files and command behavior.
  • avoid brittle coefficient snapshots and leakage-prone evaluation.

Three useful layers​

  1. Data assumptions: required columns exist; IDs are unique; target values are allowed; declared holdout entities exist; numeric values parse; both partitions contain both classes.
  2. Training behavior: the CLI exits successfully from declared inputs and publishes the documented artifact schemas.
  3. Model behavior: the known holdout contains eight predictions; accuracy is 0.875; log loss is 0.159471; ROC AUC is 1.0; a repeated run preserves the artifact bytes in the verified environment.

The literal expected values come from the reviewed teaching fixture and are independent acceptance facts. A test that recomputes its expected score using the same code path would be tautological.

Worked example​

Run the public verifier rather than importing trainer internals:

npm run test:ml-engineering-reproducibility

The verifier starts the CLI in two fresh temporary output directories, reads its public JSON/CSV artifacts, and compares them with documented values. It also checks that the data and config hashes in run.json identify the actual files. The test remains valid if the script is reorganized into packages because it observes only the public command and artifacts.

A negative contract matters too. Duplicated student IDs, an absent holdout ID, a non-binary target, or a partition with one class should stop training before artifacts are presented as successful. In larger systems, validate ranges, units, time windows, missingness, referential integrity, category evolution, and leakage-sensitive availability.

What not to freeze​

Exact coefficients are useful artifact content but usually a brittle primary test. A legitimate solver or dependency update can change low-order digits while preserving model behavior. Compare metrics and predictions with tolerances appropriate to the risk, then separately review intended environment changes. Conversely, a broad tolerance must not hide a changed class mapping or broken threshold.

Testing cannot prove fairness, future generalization, causal validity, or safety. It can make specific assumptions executable and expose drift quickly. Human review and production monitoring remain necessary.

Common mistakes​

  • Unit-testing private preprocessing helpers while never running the public training command.
  • Mocking project-owned pipeline steps, so integration failures disappear.
  • Testing that a metric is between zero and one but not that it matches a known fixture.
  • Fitting preprocessing on all rows before splitting, creating data leakage.
  • Using the final holdout to tune assertions repeatedly.

Practice​

Add a hypothetical country feature. Write one data assumption, one leakage risk, and one observable behavior test. Do not name a private function. Explain which failure should block training and which should trigger a model review.

Next steps​

Reproducible Training Pipelines composes validated stages so the accepted order is executable rather than remembered.

Primary sources​

Checked 2026-10-15: scikit-learn common pitfalls, scikit-learn metrics and scoring, Python subprocess documentation, and pytest good integration practices.