Skip to main content

Nonlinear Optimization

Nonlinear models allow curved objectives and constraints, but curvature changes what solver termination means. A local solver can converge successfully to a point that is worse than another feasible local optimum.

Prerequisites​

Complete constraint programming and review convexity and local versus global optima.

Learning goals​

  • Declare a continuous variable, bounded domain, nonlinear objective, and units.
  • Distinguish solver convergence from global optimality.
  • Use multiple starts as a diagnostic rather than a proof.
  • Check bounds, stationarity, objective values, and sensitivity independently.

Formulation before solver choice​

Let x∈[−2,2]x\in[-2,2] be a standardized policy adjustment measured in standard-deviation units. Minimize the illustrative loss

f(x)=(x2−1)2+0.2xf(x)=(x^2-1)^2+0.2x

in dimensionless loss points. The quartic term creates two wells, and the linear term makes the negative well better. The box is the only constraint. This is a repository-owned mathematical instance, not an empirical policy model.

SciPy's bounded L-BFGS-B is a local derivative-based method. Its success flag and status describe algorithm termination from one initial point. They do not certify the best point over a non-convex domain.

Worked example​

from scipy.optimize import minimize

def loss(vector):
x = vector[0]
return (x * x - 1.0) ** 2 + 0.2 * x

results = []
for start in (-0.8, 0.8):
result = minimize(
loss,
[start],
method="L-BFGS-B",
bounds=[(-2.0, 2.0)],
options={"ftol": 1e-12, "gtol": 1e-10, "maxiter": 1000},
)
results.append((start, result))
print(f"start={start:+.1f} success={result.success} status={result.status} x={result.x[0]:+.4f} loss_points={result.fun:+.6f}")

best = min(results, key=lambda pair: pair[1].fun)[1]
print(f"best_observed_x={best.x[0]:+.4f} best_observed_loss={best.fun:+.6f} global_proof=False")

Expected output:

start=-0.8 success=True status=0 x=-1.0241 loss_points=-0.202440
start=+0.8 success=True status=0 x=+0.9740 loss_points=+0.197434
best_observed_x=-1.0241 best_observed_loss=-0.202440 global_proof=False

Both runs converge successfully, yet the positive-well solution is worse. Multi-start exposes that local dependence but supplies no global proof. For this one-dimensional teaching case, the verifier independently solves f′(x)=4x3−4x+0.2=0f'(x)=4x^3-4x+0.2=0, evaluates every real stationary point and both bounds, and confirms the negative well is globally best. That exhaustive stationary-point argument does not generalize automatically to high dimensions.

Limitations and common mistakes​

  • success=True means the selected algorithm met its termination condition, not that it found a global optimum.
  • Multi-start can miss narrow or disconnected basins and provides no general global certificate.
  • Finite-difference derivatives, scaling, noisy objectives, and loose tolerances can change termination.
  • Bounds in standardized units must map back to operationally meaningful limits.
  • A mathematically global optimum of a misspecified loss remains a poor real decision.

Practice​

Run starts from -2 to 2 in steps of 0.25, cluster the converged points, and compare each with the stationary roots. Then change the linear coefficient from 0.2 to -0.2 and predict which well becomes better before solving.

Next steps​

Ticket 60 continues with stochastic optimization, which makes uncertain outcomes and recourse explicit before moving through robust and simulation-based methods, solver evidence, validation, predictive-to-prescriptive boundaries, and the constrained-delivery project.

Primary sources​

Checked 2026-10-19: SciPy 1.18.1, minimize interface and status contract, minimize(method='L-BFGS-B') options, and the optimization tutorial on local and global methods; Boyd and Vandenberghe, Convex Optimization.