Batch vs Online Inference

Inference is the work that turns validated features and a fixed model version into predictions. The same mathematical prediction can be delivered by a scheduled file job or inside a request path, but those delivery modes create different operational obligations.

Prerequisites

Complete Model Registry and Lineage. Know the Study Sprint model artifact, its input features, and why an immutable version identifies more than a filename.

Learning goals

• choose batch or online inference from a decision's deadline rather than fashion.
• connect freshness, latency, throughput, availability, and cost to a workload.
• keep one prediction contract and preprocessing meaning across both paths.

Start with the decision clock

Use batch inference when predictions can be computed before consumption: a nightly study-support list, a weekly capacity plan, or backfilling a new model over historical rows. Batch makes large sequential reads, retries, reconciliation, and cost scheduling easier. Its useful freshness is bounded by source arrival, job start, runtime, and publication delay.

Use online inference when an application needs a prediction after request-specific facts arrive and before it can respond. Online service introduces a latency budget, concurrent demand, load shedding, deployment overlap, and an availability dependency in the caller's path. It does not automatically make underlying features fresh.

Streaming is an input or computation pattern, not a synonym for online prediction. A stream can feed a later batch, and an online endpoint can read slowly refreshed state.

Worked example

Study Sprint needs two decisions:

Decision	Delivery	Reason
prepare tomorrow's optional tutor queue	nightly batch	all eligible learners can be scored together; retry before staff arrive
choose an in-session hint	online	the latest practice state is known only during the session

Both send the same versioned request shape and must produce the same result for the same feature values. The repository demonstrates this invariant with:

python examples/ml-engineering/predict.py batch \
  --model build/study-sprint/model.json \
  --input fixtures/ml-engineering/inference-requests.jsonl \
  --output build/study-sprint/predictions.jsonl

The command is local and deterministic. A real batch design must additionally define input snapshot identity, output partition ownership, idempotent replacement, retry policy, and reconciliation counts.

Workload tradeoffs

Do not copy a universal “online is under N milliseconds” target from toy hardware. Derive a budget from the user action, then allocate it among network, feature retrieval, validation, model computation, and downstream work. For batch, derive a completion deadline and required records per unit time. Measure the real model, payload mix, concurrency, warm-up, and failure modes on representative infrastructure.

Availability is also contextual. A recommendation can sometimes disappear safely; an eligibility decision may need a reviewed fallback or fail closed. Batch may tolerate a retry within its publication window while an online caller may have only tens or hundreds of milliseconds left.

Common mistakes

• Calling a nightly table “real time” because the final lookup is fast.
• Selecting online serving before defining a request-time decision.
• Maintaining separate feature transformations for batch and online scoring.
• Comparing one-row latency with bulk throughput as if they were interchangeable.
• Ignoring how late input, retries, and partial batch publication affect consumers.

Practice

Choose batch or online delivery for three Study Sprint actions: next-week staffing, a session hint, and an end-of-day learner email. For each, state the decision deadline, permissible staleness, failure behavior, volume shape, and owner. A hybrid answer is allowed, but identify which path is authoritative.

Next steps

Inference Contracts and Schema Validation defines the shared request, response, version, and error boundary before either path is implemented.

Primary sources

Checked 2026-10-15: Google SRE on embracing risk and distinct latency/throughput workloads, Google SRE service best practices, and scikit-learn model-serving guidance.