Shadow, Canary, and Blue–Green Rollouts
A rollout strategy controls exposure to change. It does not establish that a model is valid, fair, secure, or beneficial. Define what receives traffic, what can affect users, what is measured, and who decides promotion or rollback.
Prerequisites
Complete Offline–Online Feature Consistency. Have immutable Study Sprint model versions, compatible request contracts, and observable service outcomes.
Learning goals
- distinguish shadow, canary, rolling, and blue–green delivery.
- choose assignment units and guardrails that match the failure risk.
- keep rollback executable without assuming it reverses completed decisions.
Four patterns
Shadow: copy selected requests to a candidate, discard its decisions, and compare safely. Shadowing tests candidate behavior under live-shaped traffic without direct user impact, but doubles some work, can expose sensitive data, and may trigger side effects unless the shadow path is strictly read-only. Delayed labels may prevent immediate quality comparison.
Canary: route a small, defined share of decision-making traffic to the candidate, monitor guardrails and outcomes, then expand or roll back. Random request sampling is not always appropriate: assign persistently by learner/account when cross-request consistency matters, and account for selection and interference in analysis.
Blue–green: keep two complete environments, validate the idle candidate (“green”), switch traffic from current (“blue”), and retain a rapid switch-back path. It costs overlapping capacity and still needs data/schema compatibility.
Rolling: replace instances gradually inside one serving pool. It limits simultaneous replacement but can mix versions during the rollout and is not automatically a statistically interpretable canary.
Worked example
Study Sprint proposes study-sprint-completion@2 with the same request schema:
- replay a fixed offline corpus and contract suite;
- shadow a privacy-reviewed sample, with no learner-facing output;
- compare errors, latency, feature missingness, score/calibration evidence, and slices when labels mature;
- canary by a stable learner assignment, not changing version on each request;
- expand only after a declared observation window and owner review;
- keep version 1 and its compatible feature path ready until rollback risk expires.
Promotion criteria must be set before observing results. Include service error/latency guardrails and decision-quality or harm measures appropriate to the use. “No alert fired” is not a complete success criterion.
Rollback boundaries
Rollback redirects future requests. It cannot retract a sent message, undo a denied action, restore deleted state, or repair corrupted features. If the candidate writes data, use idempotency, versioned outputs, migration compatibility, and a forward-recovery plan. Rehearse authority, commands, verification, and communication.
Shadow and canary traffic must honor privacy, consent, retention, deletion, and access policy. Duplicate processing is still processing. Keep model and assignment identifiers for analysis without storing unnecessary raw features.
Common mistakes
- Calling a mixed rolling update a controlled canary without stable assignment.
- Letting shadow predictions trigger downstream side effects.
- Comparing candidates on different feature definitions or freshness.
- Scaling a rollout before delayed outcome evidence can arrive.
- Assuming rollback makes every consequence disappear.
Practice
Write a Study Sprint version-2 rollout plan with entry evidence, shadow sample, stable canary unit, stages, capacity during overlap, metrics, minimum observation window, stop thresholds, approver, rollback steps, and irreversible effects.
Next steps
Fallbacks, Timeouts, and Degraded Modes defines behavior when the selected service or its dependencies cannot answer correctly in time.
Primary sources
Checked 2026-10-15: Kubernetes canary deployment tutorial, Kubernetes Deployment behavior and rollback, Google SRE service rollout practices, and Google SRE Workbook on canarying releases.