Validating Optimization Models
Verification asks whether the implementation solves the model intended. Validation asks whether that model is useful for the decision context. A successful solver call answers neither question on its own.
Prerequisites
Complete solver statuses, gaps, and time limits.
Learning goals
- Test constraint satisfaction and operational invariants independently from the solver objective.
- Recompute objective components from returned decisions.
- Use hand cases, boundary cases, infeasible cases, and out-of-sample stress cases.
- Keep mathematical verification distinct from operational and external validity.
A layered validation plan
- Data contract: check schema, units, ranges, identities, probability sums, and missingness before building the model.
- Formulation checks: trace every objective coefficient and constraint to a declared assumption; inspect dimensions and signs.
- Solver checks: retain status, gap, tolerances, and any incumbent/bound.
- Independent recomputation: calculate resource use, integrality, constraint residuals, and objective components without trusting the solver's aggregate objective.
- Behavioral cases: test an obvious tiny solution, a binding boundary, a deliberate contradiction, and monotonic changes where theory supports them.
- Operational validation: replay held-out or shadow-mode cases, inspect fallback use and failures, and obtain domain-owner review.
An operational invariant can be absent from the optimizer yet still block deployment—for example, “no more than three packages can use the courier fallback.” Such checks must not disappear merely because the modeled objective is low.
Worked example
scenarios = [(0.2, 4, 4), (0.5, 7, 6), (0.3, 9, 8)]
route_a, route_b = 2, 1
driver_hours = 5 * route_a + 4 * route_b
depot_slots = route_a + route_b
east_capacity = 4 * route_a + route_b
west_capacity = route_a + 4 * route_b
violated = []
if driver_hours > 14:
violated.append("driver_hours")
if depot_slots > 3:
violated.append("depot_slots")
binding = sorted(name for name, used, limit in
[("driver_hours", driver_hours, 14), ("depot_slots", depot_slots, 3)]
if used == limit)
scenario_costs = []
for _, east, west in scenarios:
shortage = max(0, east - east_capacity) + max(0, west - west_capacity)
scenario_costs.append(80 * route_a + 70 * route_b + 50 * shortage)
recomputed = sum(row[0] * cost for row, cost in zip(scenarios, scenario_costs))
print(f"integral={all(isinstance(value, int) for value in (route_a, route_b))} driver_hours={driver_hours}/14 depot_slots={depot_slots}/3")
print(f"violated={violated} binding={binding}")
print(f"recomputed_expected_cost_eur={recomputed:.2f} solver_objective_match={recomputed == 260}")
Expected output:
integral=True driver_hours=14/14 depot_slots=3/3
violated=[] binding=['depot_slots', 'driver_hours']
recomputed_expected_cost_eur=260.00 solver_objective_match=True
The objective matches and both resource limits bind. Those facts are tested separately: matching EUR 260 would not excuse a violated hard constraint.
Limitations and common mistakes
- Testing with the same formula copied from implementation can reproduce the same bug; use independently derived literals or enumeration for tiny cases.
- Historical replay is optimistic if the model or thresholds were selected on those same cases.
- Feasibility under nominal data can hide fragile near-boundary behavior.
- Stakeholder sign-off does not replace quantitative checks, and quantitative checks do not establish causal or external validity.
Practice
Change the plan to three route-A runs. Confirm that depot slots remain feasible but driver-hours fail. Then add a unit test whose failure names the operational invariant, not an internal function.
Next steps
Apply the validation layers across a learned input boundary in combining predictive models with optimization.
Primary sources
Checked 2026-10-19: SciPy 1.18.1, milp result and constraint interface; HiGHS, terminology and feasibility concepts and official documentation; Google OR-Tools, solver status contract.