Gradient Boosting Theory

Gradient boosting constructs a function $F$ that minimizes empirical risk

$$\mathcal{R}(F)=\sum_{i=1}^{n}L(y_i,F(x_i)).$$

Unlike ordinary parameter gradient descent, it searches in function space using a restricted family of base learners.

1. Stage-wise additive optimization

At stage $m$, start with $F_{m-1}$ and compute the negative gradient at every training observation:

$$r_{im}=-\left.\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right|_{F=F_{m-1}}.$$

Fit a base learner $h_m$ to $(x_i,r_{im})$. A line search can then choose

$$\rho_m=\arg\min_{\rho}\sum_i L\left(y_i,F_{m-1}(x_i)+\rho h_m(x_i)\right),$$

and update $F_m(x)=F_{m-1}(x)+\nu\rho_m h_m(x)$. Implementations may optimize leaf values directly rather than perform one global line search.

2. Loss-specific pseudo-residuals

For $L=(y-F)^2/2$, $r=y-F$: ordinary residuals. For binary logistic loss with raw score $F$ and $p=\operatorname{sigmoid}(F)$, $r=y-p$. The ensemble's additive output is therefore a raw margin; a link function converts it to a probability.

Absolute-error and quantile losses are not differentiable everywhere, but subgradients and leaf-wise updates make robust regression possible. The choice of loss changes both what is optimized and what the output means.

3. Why trees work as base learners

A regression tree partitions feature space and gives each leaf a constant update. Depth-one trees express main effects; deeper trees can represent interactions. In a $J$-leaf tree, the update is

$$h_m(x)=\sum_{j=1}^{J}b_{jm}\,\mathbf{1}(x\in R_{jm}).$$

Tree construction is greedy. Boosting does not make the resulting optimizer globally optimal, and training loss decreasing at each stage does not prove generalization.

4. Regularization mechanisms

Shrinkage ($0<\nu\le1$) reduces each stage's contribution and usually requires more rounds. Shallow trees, minimum leaf sizes, row/column subsampling, and explicit penalties reduce capacity in different ways. Their effects are coupled; tune coherent ranges rather than one parameter at a time while freezing arbitrary defaults.

Early stopping is also model selection. The validation sequence influences the selected iteration, so repeated inspection can overfit that validation set. Preserve a test set or use an outer cross-validation loop.

The controls regularize different parts of the procedure. Tree depth or leaf count limits the interactions available in one stage; minimum leaf support limits how narrowly a stage can specialize; shrinkage limits the contribution of every new tree; subsampling changes the data or features visible to a stage; and the round count controls how long the additive fit continues. One setting does not substitute mechanically for another. For example, a smaller learning rate commonly needs more rounds, but it does not make an excessively flexible leaf structure harmless.

The training-loss path is expected to improve as stages are added, while validation loss may flatten or deteriorate. The selected stopping round is therefore a result of the particular training/validation split and objective. Record that round and examine its stability across suitable resamples instead of interpreting it as an intrinsic property of the dataset.

5. Second-order variants

XGBoost and related libraries use a second-order approximation around the current predictions:

$$\mathcal{L}^{(m)}\approx\sum_i\left[g_i f_m(x_i)+\frac12 h_i f_m(x_i)^2\right]+\Omega(f_m),$$

where $g_i$ and $h_i$ are first and second derivatives of the loss. The Hessian here is loss curvature, not a sample count; parameters such as XGBoost's min_child_weight constrain sums of Hessians.

For a fixed tree structure, sums of gradients indicate the direction and magnitude of a useful leaf update, while sums of Hessians scale that update by local curvature. Regularization shrinks or suppresses leaves whose evidence is weak. Libraries differ in their exact gain formulas, growth policies, binning, and categorical algorithms, so similarly named parameters are not guaranteed to define equivalent models.

6. Statistical interpretation and limits

Boosted trees are powerful predictors of nonlinearities and interactions, but they do not make observational relationships causal. Missing-value routing and native categorical algorithms learn from training data and must be applied with the same schema at inference. High-cardinality identifiers, time order, repeated entities, and target-derived features can leak information even when the library API is used correctly.

Use group-aware or time-aware splits where the deployment boundary requires them. Compare against simple baselines and report uncertainty across appropriate resamples rather than a single favorable split.

Summary

Gradient boosting fits negative loss gradients with a constrained learner and adds regularized steps to an ensemble. Squared-error residual fitting, second-order tree construction, and early stopping are special mechanisms within that broader framework; none removes the need for leakage-safe validation.