Skip to main content

Project: Train, Register, Deploy, and Monitor

This capstone composes the public seams from Tickets 39–41. It trains the deterministic Study Sprint classifier, registers immutable versions, evaluates promotion criteria, exercises batch serving, consumes a monitoring signal, rolls back the production alias, validates the fallback, and writes a model card. It uses files as the smallest sufficient platform.

Prerequisites​

Complete Build vs Buy and Platform Costs and all earlier ML Engineering & MLOps lessons. Install the repository's pinned content-verification dependencies; the run itself needs no secret, cloud account, hosted service, or network access.

Learning goals​

  • execute one repeatable train–register–deploy–monitor lifecycle through public CLIs.
  • gate promotion on explicit evidence and keep alias movement separate from artifact identity.
  • prove monitoring-to-rollback behavior and package operational documentation.

Worked example​

From a clean checkout and verified Python environment, run:

python examples/ml-engineering/capstone.py \
  --output-dir build/study-sprint-capstone \
  --code-version reviewed-commit

The command publishes only after every phase succeeds:

  1. train.py produces the inert model, metrics, predictions, and lineage record.
  2. Four promotion criteria require accuracy at least 0.85, log loss at most 0.2, ROC AUC at least 0.9, and the literal eight-row holdout.
  3. registry.json records immutable version identities and artifact digests; the production alias moves independently.
  4. predict.py batch verifies the versioned serving contract and literal probabilities 0.015377 and 0.996779.
  5. monitor.py processes seven local windows; the final concept-drift scenario emits rollback-and-investigate with owner and runbook.
  6. The alias returns from study-sprint-completion@1 to study-sprint-completion@0, and the fallback artifact passes the same batch contract.
  7. model-card.md and lifecycle-report.json preserve purpose, evidence, limitations, monitoring, and rollback.

Run the batch verifier:

npm run test:ml-engineering-platform

It executes the capstone twice in clean temporary directories, compares every file digest, and independently asserts lineage, gates, predictions, monitoring, registry state, rollback, documentation, source review, and ledger rows. The other three ML Engineering verifiers remain independent acceptance seams.

The @0 fallback is a synthetic local copy with a separate version identity, used only to prove artifact compatibility and alias rollback. It is not historical production evidence. Likewise, the tiny fixture and fixed thresholds prove deterministic mechanics—not model value, external validity, fairness, privacy, or production readiness.

Common mistakes​

  • Promoting because training exited zero without checking declared criteria.
  • Overwriting model bytes under a mutable version or filename.
  • Deploying a candidate without a compatible previous artifact and rollback test.
  • Retraining automatically when monitoring says investigate or roll back.
  • Publishing metrics without cohort, lineage, limitations, owner, and runbook.

Practice​

Change one promotion threshold so the candidate fails. Predict which artifacts should remain unpublished. Then design a canary-only signal that pauses promotion without rolling back the current production version.

Next steps​

The ML Engineering & MLOps area is complete. Continue with Deep Learning & Modern AI, beginning with neural-network foundations.

Primary sources​

Checked 2026-10-15: MLflow Model Registry workflow, OpenLineage specification, SLSA provenance 1.2, NIST AI RMF Playbook, and Model Cards for Model Reporting.