Skip to main content

Slices, Fairness, and Harm Monitoring

An aggregate can improve while a segment is harmed. In the Study Sprint label-shift window, overall accuracy is 0.7, but the reported slice accuracies are 1.0 for advanced and 0.4 for foundations. The 0.6 gap is a prompt to investigate—not a complete fairness judgment.

Prerequisites​

Complete Production Performance and Calibration. Bring the model card's intended users, exclusions, known risks, and evaluation slices.

Learning goals​

  • choose slices from plausible failure mechanisms and affected communities.
  • distinguish metric disparity, uncertainty, allocation, and experienced harm.
  • define escalation and remedy with domain and community input.

Worked example​

For each slice, report traffic and decision counts, label coverage and delay, score and threshold distributions, confusion-matrix rates, calibration, overrides, appeals, and relevant outcomes. Keep denominators visible. Small groups need uncertainty intervals and sometimes longer windows; hiding them entirely can erase the people most at risk.

Fairness criteria answer different questions and can conflict. Demographic parity concerns selection rates. Equalized-odds components condition error rates on outcomes. Calibration compares predicted probabilities with observed frequencies within groups. Which evidence matters depends on the decision, base rates, measurement, access, remedy, and legal and ethical context. Passing one metric does not prove fairness; a disparity does not by itself identify its cause.

Monitor harms beyond prediction error:

  • allocation: denial or unequal access to resources;
  • quality of service: worse latency, fallbacks, or prediction quality;
  • representation and dignity: stereotyping, stigmatizing labels, or inappropriate inferences;
  • process: inability to contest, understand, or obtain human review;
  • feedback: decisions change future data, opportunity, and visibility.

Sensitive attributes can be necessary to audit disparities but create privacy, security, consent, and governance duties. Do not infer protected identity casually. Restrict access, retention, and use; document missing or unknown values instead of forcing a category.

When harm signals rise, protect users first: pause a use, widen human review, restore a previous approved policy, or provide remedy. Then investigate data, measurement, threshold, workflow, and institutional causes. Retraining is only one possible response.

Common mistakes​

  • Choosing slices only after seeing a favorable result.
  • Comparing percentages without counts, coverage, or uncertainty.
  • Treating fairness as one universal parity equation.
  • Optimizing model metrics while ignoring access, appeals, and downstream policy.
  • Logging protected data without a governed purpose.

Practice​

Extend the Study Sprint model card with two predeclared slices and one intersection. For each, name a plausible harm, evidence, minimum support, owner, privacy control, safe response, and the people who should participate in review.

Next steps​

Continue with Alerts, Dashboards, and Runbooks to route this evidence without creating alert fatigue.

Primary sources​

Checked 2026-10-15: NIST SP 1270 on bias in AI, NIST AI RMF 1.0, Fairlearn fairness assessment guide, and Model Cards for Model Reporting.