Latency, Throughput, and Cost
Serving performance is a distribution under a workload, not one stopwatch number. A system can improve average latency while worsening tail latency, increase throughput by batching while delaying sparse requests, or reduce compute cost while making failures more expensive.
Prerequisites
Complete Containerizing Inference. Know the difference between batch completion deadlines and online response budgets.
Learning goals
- define latency, throughput, utilization, queueing, availability, and cost in caller terms.
- construct a representative measurement plan without universal toy numbers.
- recognize tradeoffs among batching, concurrency, capacity headroom, and freshness.
Define the workload first
For Study Sprint's online hint, describe request arrival over time, instances per request, payload distribution, feature-cache hit rate, model version, warm/cold state, concurrency, and downstream deadline. For the nightly tutor queue, describe row count, input partitions, completion deadline, retry window, output atomicity, and competing workloads.
Measure at the user-visible boundary. Useful summaries include request success rate and latency percentiles over a declared window, not only a mean. Throughput is completed useful work per unit time at an acceptable error and latency level. Offered load is not completed throughput. Once demand exceeds service capacity, queues usually raise latency before failures make overload obvious.
Worked example
Create a measurement table before running any benchmark:
| Question | Declared Study Sprint experiment |
|---|---|
| workload | replay a fixed local request corpus with realistic batch sizes |
| variants | model version, process count, batch size, warm/cold start |
| measures | successful instances/s, p50/p95/p99 latency, errors, CPU, memory |
| limits | one named machine/runtime; loopback excludes network and feature retrieval |
| decision | smallest configuration meeting the declared deadline with headroom |
Do not publish its numbers as general guidance. Repeat long enough to observe warm-up and variance, report confidence or repeated-run spread, and retain workload plus environment metadata. Test overload and recovery, not just a comfortable steady state.
Tradeoffs
Larger batches amortize parsing and vectorized computation but can wait longer to fill, use more memory, and create larger retry units. More concurrency helps until CPU, memory bandwidth, interpreter/runtime behavior, locks, or dependencies saturate. Caching can reduce latency and cost while increasing staleness and invalidation complexity. Extra replicas add availability headroom but cost money and may increase rollout overlap.
Cost includes compute, storage, network, observability, engineering, incident response, and opportunity cost. Compare alternatives at the same workload and service objective. A cheaper response that violates freshness or causes harmful fallback decisions is not equivalent work.
Capacity and availability
Choose headroom for bursts, failures, maintenance, and rollouts. Load-test resource exhaustion and queue behavior in a safe environment. A service should reject excess work early or offer a defined degraded response rather than accept an unbounded queue that converts overload into timeouts.
Common mistakes
- Reporting only average latency.
- Equating requests per second with predictions per second when batch sizes differ.
- Benchmarking random toy inputs that miss expensive real shapes.
- Measuring a warm model and planning no cold-start or rollout capacity.
- Ignoring feature retrieval, serialization, network, and caller timeout.
Practice
Write separate measurement plans for the nightly Study Sprint batch and the in-session endpoint. State the decision each benchmark supports, controlled variables, workload fixture, percentile/window, overload point, cost boundary, and why the results cannot be generalized beyond that environment.
Next steps
Offline–Online Feature Consistency protects semantic correctness when performance pressure tempts teams to duplicate or simplify feature logic.
Primary sources
Checked 2026-10-15: Google SRE on service best practices and representative load testing, Google SRE on overload and queues, Google SRE on availability as successful requests, and Google SRE on capacity, efficiency, and latency.