Nonlinear Optimization
Nonlinear models allow curved objectives and constraints, but curvature changes what solver termination means. A local solver can converge successfully to a point that is worse than another feasible local optimum.
Prerequisites
Complete constraint programming and review convexity and local versus global optima.
Learning goals
- Declare a continuous variable, bounded domain, nonlinear objective, and units.
- Distinguish solver convergence from global optimality.
- Use multiple starts as a diagnostic rather than a proof.
- Check bounds, stationarity, objective values, and sensitivity independently.
Formulation before solver choice
Let be a standardized policy adjustment measured in standard-deviation units. Minimize the illustrative loss
in dimensionless loss points. The quartic term creates two wells, and the linear term makes the negative well better. The box is the only constraint. This is a repository-owned mathematical instance, not an empirical policy model.
SciPy's bounded L-BFGS-B is a local derivative-based method. Its success flag and status describe algorithm termination from one initial point. They do not certify the best point over a non-convex domain.
Worked example
from scipy.optimize import minimize
def loss(vector):
x = vector[0]
return (x * x - 1.0) ** 2 + 0.2 * x
results = []
for start in (-0.8, 0.8):
result = minimize(
loss,
[start],
method="L-BFGS-B",
bounds=[(-2.0, 2.0)],
options={"ftol": 1e-12, "gtol": 1e-10, "maxiter": 1000},
)
results.append((start, result))
print(f"start={start:+.1f} success={result.success} status={result.status} x={result.x[0]:+.4f} loss_points={result.fun:+.6f}")
best = min(results, key=lambda pair: pair[1].fun)[1]
print(f"best_observed_x={best.x[0]:+.4f} best_observed_loss={best.fun:+.6f} global_proof=False")
Expected output:
start=-0.8 success=True status=0 x=-1.0241 loss_points=-0.202440
start=+0.8 success=True status=0 x=+0.9740 loss_points=+0.197434
best_observed_x=-1.0241 best_observed_loss=-0.202440 global_proof=False
Both runs converge successfully, yet the positive-well solution is worse. Multi-start exposes that local dependence but supplies no global proof. For this one-dimensional teaching case, the verifier independently solves , evaluates every real stationary point and both bounds, and confirms the negative well is globally best. That exhaustive stationary-point argument does not generalize automatically to high dimensions.
Limitations and common mistakes
success=Truemeans the selected algorithm met its termination condition, not that it found a global optimum.- Multi-start can miss narrow or disconnected basins and provides no general global certificate.
- Finite-difference derivatives, scaling, noisy objectives, and loose tolerances can change termination.
- Bounds in standardized units must map back to operationally meaningful limits.
- A mathematically global optimum of a misspecified loss remains a poor real decision.
Practice
Run starts from -2 to 2 in steps of 0.25, cluster the converged points, and compare each with the stationary roots. Then change the linear coefficient from 0.2 to -0.2 and predict which well becomes better before solving.
Next steps
Ticket 60 continues with stochastic optimization, which makes uncertain outcomes and recourse explicit before moving through robust and simulation-based methods, solver evidence, validation, predictive-to-prescriptive boundaries, and the constrained-delivery project.
Primary sources
Checked 2026-10-19: SciPy 1.18.1, minimize interface and status contract, minimize(method='L-BFGS-B') options, and the optimization tutorial on local and global methods; Boyd and Vandenberghe, Convex Optimization.