Fallbacks, Timeouts, and Degraded Modes
An inference dependency will eventually be slow, unavailable, overloaded, incompatible, or supplied with unusable data. Reliability comes from an explicit decision for each failure class, not from retrying everything.
Prerequisites
Complete Shadow, Canary, and Blue–Green Rollouts. Know the Study Sprint request deadline, caller behavior, and stable error contract.
Learning goals
- allocate an end-to-end deadline instead of stacking independent default timeouts.
- retry only bounded, transient, safe work.
- distinguish fallback, fail-open, fail-closed, stale response, and degraded behavior.
Deadlines and cancellation
Start with the caller's usable response deadline. Reserve time for its own recovery, then pass a smaller deadline through feature retrieval and inference. A timeout limits how long the caller waits; cancellation should stop obsolete work where supported. Without it, timed-out work can continue consuming capacity and amplify overload.
Connect and read timeouts protect different phases. Warm and cold paths may differ. Measure the dependency distribution and false-timeout cost on representative workloads; do not copy one universal number.
Retry budget
Retry only failures likely to be transient and only when the operation is safe to repeat. Use a small attempt limit, remaining-deadline check, backoff, and jitter. Put retries at one layer when possible: three attempts at each of three nested layers can multiply load. Do not retry malformed, out-of-domain, or version-mismatched Study Sprint requests; the same input will fail again.
Inference reads are often side-effect-free, but the overall decision path may not be. Use idempotency keys before retrying any action that writes or sends. A retry budget should be visible in load and availability planning.
Worked example
For an optional Study Sprint hint:
| Failure | Behavior | Reason |
|---|---|---|
| malformed or missing feature | stable 4xx; no retry | producer must repair its contract |
| requested model version unavailable | 409; caller follows declared compatibility policy | silently substituting changes meaning |
| brief internal transport fault | at most one retry inside remaining deadline | possible transient failure with bounded amplification |
| deadline exhausted | omit personalized hint and show standard study guidance | the feature is optional and the fallback is reviewed |
| stale feature | reject or use a marked last-known value only if policy permits | low lookup latency does not make stale data current |
That degraded response is specific to a low-stakes optional hint. A safety, access, credit, medical, or legal decision could require fail-closed behavior, escalation, or human review. “Use the old model” is safe only if its contract, features, vulnerabilities, and intended population remain supported.
Load shedding and fallbacks
Bound queues and reject excess work early before latency consumes every caller's deadline. A degraded mode should be cheaper, independently tested, observable, and semantically honest. Cached predictions need identity, age, expiry, and invalidation rules; never present them as fresh silently.
Track fallback rate, reason, duration, affected population, and recovery. A service that returns 200 from a fallback may be technically available while the intended ML capability is unavailable, so measure both.
Common mistakes
- Retrying deterministic validation errors.
- Giving each dependency the full user deadline.
- Layering retries without an overall attempt or time budget.
- Serving stale or default values without marking them.
- Choosing fail-open or fail-closed without evaluating consequence and reversibility.
Practice
Create a failure-mode table for a Study Sprint feature lookup, model loader, prediction calculation, and downstream hint publisher. For each, decide detection, retryability, deadline, fallback, user-visible meaning, metric, and owner.
Next steps
Project: Serve a scikit-learn Model combines the artifact, shared contract, batch path, loopback API, and integration tests.
Primary sources
Checked 2026-10-15: Amazon Builders' Library on timeouts, retries, and jitter, Google SRE on handling overload, Google SRE on preventing cascading failures, and RFC 9110 client and server error semantics.