Production Performance and Calibration

One metric cannot describe every use of a score. Ranking, probability estimation, and a thresholded Study Sprint decision have different failure modes, denominators, and costs.

Prerequisites

Complete Monitoring with Delayed Labels. Use only labels from a declared mature and sufficiently representative cohort.

Learning goals

• separate ranking, probability, threshold, and outcome metrics.
• interpret calibration relative to population, time, and intended use.
• compare production evidence with a versioned evaluation reference.

Worked example

The reference window has accuracy 0.8, Brier score 0.2005, and a two-bin expected calibration error (ECE) of 0.09. Under harmful concept drift, accuracy is 0.2, Brier score is 0.5005, and ECE is 0.51. The same probability vector was used, so the change comes from the observed input–outcome relationship, not score movement.

The Brier score for binary outcomes is

$$BS=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2,$$

so lower is better for the defined cohort. It combines calibration and refinement; it should not be described as calibration alone. A reliability diagram groups predictions and compares mean prediction with observed frequency. ECE summarizes those bins but depends on binning, sample size, and weighting and can hide local errors.

Track according to use:

• ranking: ROC AUC or precision–recall behavior with class balance visible;
• probabilities: log loss, Brier score, and reliability plots;
• decisions: sensitivity, specificity, precision, recall, error cost, abstention, and capacity at the deployed threshold;
• outcomes: user benefit, failure, override, appeal, and adverse consequence with a credible attribution design.

A threshold can become inappropriate after prevalence or cost changes even when ranking remains stable. Recalibration can repair some probability mismatch but cannot repair missing features, an invalid target, or harmful product policy. Estimate uncertainty and minimum sample requirements; do not page on single-event metric jumps.

Common mistakes

• Calling AUC a calibration metric.
• Comparing accuracy across windows without prevalence and decision coverage.
• Computing performance on selected labels as if complete.
• Recalibrating and claiming concept drift is solved.
• Tuning thresholds on production outcomes without a held-out or prospective evaluation.

Practice

Recompute the fixture's reference Brier score by hand for the first two cases, then for all ten. Choose one metric for ranking, probability quality, and threshold behavior; state its denominator and one failure it misses.

Next steps

Continue with Slices, Fairness, and Harm Monitoring so aggregate metrics do not hide concentrated failures.

Primary sources

Checked 2026-10-15: scikit-learn probability calibration guide, scikit-learn classification metrics, Brier's original probability-score paper, and NIST AI RMF 1.0.