XGBoost: In-Depth Guide

XGBoost is a scalable system for regularized tree boosting. It combines an additive objective with sparsity-aware split finding, histogram or exact tree methods, row/column sampling, and parallel work within each tree.

1. Regularized second-order objective

At boosting round $t$, XGBoost approximates the objective for a candidate tree $f_t$ as

$$\widetilde{\mathcal L}^{(t)}=\sum_i\left[g_i f_t(x_i)+\frac12h_i f_t(x_i)^2\right]+\Omega(f_t),$$

where $g_i$ and $h_i$ are the first and second derivatives of the selected loss at the current raw prediction. A common tree penalty is

$$\Omega(f)=\gamma T+\frac12\lambda\sum_{j=1}^{T}w_j^2+\alpha\sum_{j=1}^{T}|w_j|,$$

with leaf count $T$, leaf weights $w_j$, split penalty $\gamma$, and L2/L1 penalties $\lambda$ and $\alpha$. This objective explains why min_child_weight is a minimum sum of Hessians rather than a literal minimum row count.

2. Tree growth and approximation

The gain from a split follows from the gradients, Hessians, and regularization in its children. max_depth, max_leaves, min_child_weight, and gamma constrain different aspects of complexity. tree_method="hist" bins feature values for efficient approximate split search; CPU and GPU execution may use different implementations and floating-point reduction orders.

learning_rate shrinks new trees. subsample and colsample_bytree add stochastic regularization. The useful number of rounds depends on all of these settings, so it should be selected with validation rather than copied from another dataset.

3. Missing and categorical values

For a split, XGBoost learns a default direction for missing values from training data. This is not general-purpose missing-data reasoning: missingness mechanisms can change, and a production schema still needs validation.

Current XGBoost supports categorical DataFrame columns when enable_categorical=True with a compatible tree method. This support remains documented as experimental in XGBoost 3.4.2 development documentation. Persist categorical models in JSON or UBJSON, retain category dtype and levels, and benchmark native partitioning against leakage-safe one-hot encoding. Integer codes without categorical dtype are treated as numeric.

4. Objectives, outputs, and stopping

Choose an objective that matches the target: for example binary:logistic for binary probabilities or reg:squarederror for squared-error regression. The scikit-learn API returns probabilities from predict_proba, while native booster prediction has different output and iteration-range semantics.

In XGBoost 3.4, put eval_metric, early_stopping_rounds, and callbacks in the estimator constructor or set_params; they are not fit parameters. fit(..., eval_set=[...]) does not create a validation split. The scikit-learn estimator automatically uses best_iteration for prediction after early stopping, whereas native Booster.predict() needs an explicit iteration range or a callback configured to save the best model.

5. Limits

XGBoost can overfit, is sensitive to search design, and is not automatically calibrated. Split/gain importance is not causal and can be biased. Native categorical handling, GPU builds, distributed execution, and model serialization are version-sensitive; pin and validate the deployment artifact.

XGBoost is Apache-2.0 licensed. Official wheels cover common 64-bit platforms; the default package includes GPU-capable components where supported, while xgboost-cpu is a smaller CPU-only distribution.

Summary

XGBoost's distinguishing core is a regularized second-order tree objective plus a scalable systems implementation. Correct use requires objective-aware outputs, explicit category schemas, training-only early stopping, and version-specific prediction semantics.