Introduction to Gradient Boosting

Gradient boosting builds an additive prediction function one stage at a time. A new weak learner—usually a shallow regression tree—is chosen to reduce a specified loss on the training data. The method is used for regression, classification, ranking, and other objectives.

1. From boosting to gradient boosting

Bagging fits many learners independently and averages them. Boosting is sequential: learner $m$ is fitted after learners $1,\ldots,m-1$, so later learners can improve the current ensemble. “Weak” describes the deliberately limited capacity of each learner; it does not imply that every tree must be a one-split stump.

The model after $M$ stages is

$$F_M(x)=F_0(x)+\sum_{m=1}^{M}\nu\,\rho_m h_m(x),$$

where $h_m$ is a base learner, $\rho_m$ is a step size, and $\nu$ is the learning rate (shrinkage).

2. Functional gradient descent

For differentiable loss $L(y,F(x))$, stage $m$ computes pseudo-residuals

$$r_{im}=-\left[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right]_{F=F_{m-1}}.$$

The new tree is fitted to these negative gradients, not necessarily to ordinary residuals. For squared-error regression, the negative gradient is proportional to $y_i-F(x_i)$, so “fit the residuals” is exact up to a constant. For logistic loss, the model works on raw scores and the pseudo-residuals differ according to the objective's link function.

The initial constant $F_0$ also depends on the loss: it is the target mean for squared error and a class-prior log-odds value for binary logistic loss.

3. Capacity and regularization

The main controls interact:

• learning_rate and the number of boosting rounds trade smaller steps for a longer sequence;
• tree depth or leaf count controls the interaction order each stage can express;
• minimum leaf constraints and split-gain thresholds suppress weak, data-poor splits;
• row and feature subsampling add stochastic regularization;
• XGBoost, LightGBM, and CatBoost expose additional leaf-weight and tree-complexity penalties.

More rounds are not automatically harmful when shrinkage and tree capacity are controlled, but the chosen round count is still a hyperparameter. Early stopping chooses it from validation data.

4. Validation without leakage

Create an isolated test set before model selection. Use only the remaining training data for preprocessing, hyperparameter search, categorical encodings, probability calibration, threshold selection, and early stopping. A practical holdout workflow has three parts:

1. training subset for fitting tree updates;
2. validation subset for early stopping and model selection;
3. test subset used once for the final estimate.

For small data, use nested cross-validation to estimate the entire search procedure. Vendor estimators do not automatically create a leakage-safe validation set for early stopping, and passing the test set as eval_set tunes the model to the test set.

5. Choosing an objective and metrics

Match the training objective to the target and deployment question. Squared error emphasizes large regression errors; robust losses can reduce their influence. Binary log loss trains probability-like outputs, but discrimination is not calibration.

Report metrics with different roles separately:

• ROC AUC or average precision for ranking;
• log loss or Brier score for probability quality;
• confusion-matrix rates at a threshold selected on validation data;
• MAE, RMSE, or a domain-specific loss for regression.

Accuracy alone can hide class imbalance. Feature-importance rankings also require caution: split/gain importance can favor features with many candidate splits and does not establish causality. Use held-out permutation importance or carefully defined attribution methods when interpretation matters.

6. Implementations are not interchangeable

XGBoost, LightGBM, and CatBoost all build boosted trees, but differ in tree growth, categorical processing, regularization, missing-value behavior, and API semantics. None is universally fastest or most accurate. Compare them with the same data partitions, prediction target, search budget, metric, hardware, thread limit, and early-stopping policy.

All three projects are open source under the Apache License 2.0, but their Python wheels, GPU support, and platform requirements differ. Pin and test the exact library version used for an experiment or deployment.

Summary

Gradient boosting is functional gradient descent with an additive learner. Residual fitting is a squared-error special case; loss, tree capacity, shrinkage, and stopping together define the model. Fair results require training-only model selection and a genuinely isolated final evaluation.