Model Cards and Operational Documentation
A model card communicates what a model is for, how it was evaluated, where it can fail, and who owns the consequences. A runbook tells an operator what to do now. A registry record links immutable lineage and approval state. None substitutes for the others.
Prerequisites
Complete Workflow Orchestration for ML and Alerts, Dashboards, and Runbooks.
Learning goals
- document intended use, out-of-scope use, data, evaluation, slices, and limitations.
- connect a model card to exact artifact, contract, owner, monitoring, and rollback records.
- keep operational instructions testable and current.
Worked example
Ticket 42's capstone generates model-card.md from the same fixed evidence used for promotion. It records the Study Sprint version and SHA-256 digest, declared holdout metrics, synthetic-data limitation, inference contract, monitoring outcome, and rollback target. lifecycle-report.json remains the machine-readable evidence; registry.json records alias state.
A useful review packet answers:
| Question | Evidence |
|---|---|
| What decision may use this model? | Intended-use and out-of-scope statements |
| Which exact object was assessed? | Version, format, digest, code/config/data lineage |
| How was it assessed? | Locked cohort, metrics, thresholds, slices, uncertainty |
| What can fail? | Data, drift, harm, capacity, security, and feedback-loop limits |
| Who acts? | Named product, data, model, serving, risk, and incident owners |
| How is it operated? | Contract, indicators, alert rule, runbook, fallback, rollback |
Documentation should be versioned and reviewed with the artifact. Generate facts such as hashes and metrics, but require humans to own purpose, exclusions, harms, approvals, and unresolved risks. A model card that says “high accuracy” without cohort, metric definition, uncertainty, and decision context is not operational evidence.
A plain Markdown file is sufficient when one artifact and repository own the full lifecycle. A catalog or governance product is useful when discovery, access review, attestations, retention, and cross-team workflows justify its operation.
Common mistakes
- Copying a card between versions without refreshing evidence.
- Listing aggregate metrics while omitting slices, uncertainty, and label coverage.
- Writing a runbook with no owner, trigger, fallback, or recovery verification.
- Publishing sensitive training examples or incident details in documentation.
- Treating documentation as approval rather than evidence for a decision.
Practice
Review the generated Study Sprint model card. Add one out-of-scope use, one privacy limitation, one slice uncertainty, and one owner escalation without inventing evidence.
Next steps
Continue with Security, Privacy, and Supply-Chain Risk to constrain who and what can enter the lifecycle.
Primary sources
Checked 2026-10-15: Model Cards for Model Reporting, NIST AI RMF 1.0, NIST AI RMF Playbook, and Google model cards toolkit documentation.