Skip to main content

Building a Prediction API

A prediction API coordinates transport, validation, model selection, preprocessing, inference, and response encoding. The model calculation is usually the smallest part of the public responsibility.

Prerequisites​

Complete Inference Contracts and Schema Validation. Have a Ticket 39 model.json artifact and understand why only trusted, validated artifact versions may be loaded.

Learning goals​

  • map contract outcomes to stable HTTP status and JSON response behavior.
  • distinguish readiness from prediction correctness.
  • state the security and capacity boundaries of a local teaching server.

Worked example​

Train the Study Sprint artifact, then start the loopback-only service:

python examples/ml-engineering/train.py \
  --config examples/ml-engineering/training-config.json \
  --output-dir build/study-sprint \
  --code-version local-demo

python examples/ml-engineering/predict.py serve \
  --model build/study-sprint/model.json \
  --host 127.0.0.1 \
  --port 8000

From another terminal:

curl --fail-with-body \
  --header 'Content-Type: application/json' \
  --data-binary @fixtures/ml-engineering/inference-request.json \
  http://127.0.0.1:8000/v1/predict

GET /healthz reports that the process loaded study-sprint-completion@1; it does not prove dependencies, upstream features, predictive quality, fairness, or downstream health. POST /v1/predict returns 200 for a valid request, 400 for malformed JSON, 422 for a well-formed contract violation, and 409 when schema or requested model version conflicts with the loaded service state.

The handler caps the request body at 65,536 bytes and each request at 100 instances. Those are teaching limits, not measured production capacity. Batch and API call the same request validator and prediction function, so the integration test observes identical preprocessing semantics at both public seams.

Deliberate boundary​

Python documents that http.server is not recommended for production and implements only basic security checks. This example therefore rejects non-loopback bind addresses and includes no TLS, authentication, authorization, tenant isolation, distributed tracing, rate limiting, hardened parser, process supervision, or autoscaling. Do not expose it to a network by port forwarding or by weakening the host check.

A production service needs a supported server stack plus controls chosen from its threat model and SLOs. Authentication identifies a principal; authorization decides whether that principal may request this model and data. TLS protects a transport hop. None validates feature meaning or makes the model safe for a decision.

Error handling and observability​

Keep caller errors separate from server faults. Log a correlation identifier, contract/model versions, status, duration, and safe size/count metadata, but not raw sensitive features by default. Metrics should aggregate rates and latency without unbounded request IDs as labels. Internal diagnostics may be richer than the stable external envelope, subject to retention and access controls.

Common mistakes​

  • Binding a tutorial server to 0.0.0.0 and calling it deployed.
  • Returning 200 with an error embedded in a “prediction” field.
  • Loading a new model independently for every request.
  • Treating a passing health endpoint as end-to-end correctness.
  • Logging complete request bodies without a data-handling decision.

Practice​

Call the local endpoint with valid, malformed, missing-feature, out-of-range, schema-version, and model-version cases. For each, record the expected HTTP status and stable error code. Then list the controls required before any non-loopback exposure.

Next steps​

Containerizing Inference packages the command and artifact while keeping isolation, provenance, and runtime security claims honest.

Primary sources​

Checked 2026-10-15: Python http.server and its security warning, RFC 9110 status semantics, OWASP REST Security Cheat Sheet, and OpenAPI Specification.