Boosting, Part 3: Choosing a Library (XGBoost, LightGBM, CatBoost)

Boosting series: Part 1: AdaBoost and Gradient Boosting · Part 2: The XGBoost Mathematics · Part 3: Choosing a Library

Part 1 and Part 2 covered the algorithms and the XGBoost mathematics. This final part is practical: the general boosting concepts that apply to any library (feature importance, early stopping), and how to choose among XGBoost, LightGBM, and CatBoost without leaning on non-portable “fastest / most accurate” claims.

1. General Boosting Concepts

Feature Importance Boosting libraries expose several global inspection measures (not causal explanations):

• Gain: the average loss reduction a feature brings across all splits that use it.
• Frequency / Weight: how often a feature is used to split.
• Cover: the number of samples affected by its splits.

These are useful but carry well-known caveats: gain and frequency favor features with many split opportunities (high-cardinality or continuous ones); correlated predictors split or substitute each other’s importance; and the values shift with hyperparameters and random seed. For a sturdier read, prefer permutation importance on held-out data with repeated shuffles, and remember that SHAP values explain the fitted model relative to a background distribution, not a feature’s causal effect.

Early Stopping Boosting’s overfitting risk is usually controlled with early stopping: monitor a validation metric after each tree, and stop once it has not improved for a set number of rounds (the “patience”). This does not automatically find the globally optimal number of trees; it selects a round according to one validation set, one metric, and the patience and minimum-improvement rules chosen. Two disciplines matter: monitor a separate validation fold (reusing the final test set for early stopping leaks evaluation information), and evaluate once on untouched test data afterward, using the library’s recorded best_iteration.

The two curves tell the whole story. Training loss keeps dropping as trees are added, but validation loss bottoms out (here around round 175) and then creeps back up. Early stopping picks that validation minimum, not the point where training loss is smallest.

2. Comparing XGBoost, LightGBM, and CatBoost

All three are mature gradient-boosting libraries that give strong results on tabular data. They differ mostly in defaults and categorical handling, not in what they can ultimately achieve. The table below states behavior as of current stable releases - verify against the installed version, since defaults do change.

Dimension	XGBoost	LightGBM	CatBoost
Default tree growth	Depthwise (expands nodes nearest the root); lossguide (leaf-wise) also available	Leaf-wise (best-first); constrain with num_leaves / max_depth	Symmetric (oblivious) by default; Depthwise and Lossguide also available
Numeric split algorithm	Histogram (tree_method="hist", the auto default); exact and approx also available	Histogram-based	Histogram-based
Categorical features	Native since v1.5 via enable_categorical (one-hot or partition splits under hist/approx); data-type/serialization constraints apply	Native via integer category codes and gradient/Hessian statistics	Ordered target statistics for encoding plus ordered boosting to curb target leakage - two distinct mechanisms
Missing values	Learns a default direction per split	Handles NaN natively; zero is not treated as missing unless zero_as_missing=true	Configurable (separate category, or min/max)
Regularization	gamma (leaf count), lambda (L2), alpha (L1), min_child_weight, depth, row/column sampling	lambda_l1, lambda_l2, num_leaves, max_depth, min_data_in_leaf, min_gain_to_split	Symmetric structure + ordered boosting act as regularizers; plus l2_leaf_reg, depth
Notable extras	Distributed / GPU training, sparsity-aware split finding	EFB (feature bundling) on by default; GOSS sampling is opt-in, not the default gbdt booster	Built for categorical-heavy data with minimal preprocessing

Two things the older “X is fastest, Y overfits least” tables get wrong. First, those rankings are not portable: training and inference speed depend on the tree method, sparsity, thread count, device, data width, and parameters, and overfitting depends on the data and tuning, not the library name. Second, GOSS is not synonymous with LightGBM (its default booster is standard GBDT), and symmetric trees are CatBoost’s default, not a fixed property.

A Benchmark Protocol, Not a Winner

The honest way to choose is to measure on the task at hand:

1. Fix identical train/validation/test folds (time- or group-aware if the deployment is).
2. Apply leakage-safe preprocessing inside the pipeline; give each library its native categorical handling where it has one.
3. Pick a task-aligned metric: PR-AUC or a cost metric for imbalanced classification, MAE/RMSE for regression by error cost.
4. Give each a comparable tuning budget and early stopping on the validation fold.
5. Record predictive score, calibration where probabilities matter, training and inference time, and model size.
6. Evaluate once on the untouched test set.

Rules of Thumb (Not Laws)

• Categorical-heavy data with little preprocessing time: CatBoost is often the easiest starting point.
• Very large or high-dimensional data where speed and memory dominate: LightGBM’s leaf-wise histogram training is usually fastest - but constrain num_leaves / max_depth on small data, where it overfits readily.
• A dependable, well-documented default: XGBoost. Reach for the others when a specific need (categoricals, scale) argues for it.

One limitation applies to all three: like any tree ensemble, they predict leaf summaries of training targets, so a regressor does not extrapolate beyond the range of targets it saw. Model that explicitly when the target can drift outside the training range.

---

That completes the tour: from AdaBoost’s reweighting, through gradient boosting’s function-space descent and XGBoost’s regularized second-order objective, to the practical business of choosing and evaluating a library. The consistent theme is that boosting is powerful precisely because it is greedy and sequential - and that the same greed is what makes disciplined validation, honest metrics, and early stopping non-negotiable.

Resources

• Chen, T. & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. KDD ‘16.
• Ke, G., et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS 2017.
• Prokhorenkova, L., et al. (2018). CatBoost: unbiased boosting with categorical features. NeurIPS 2018.
• Official docs: XGBoost parameters, LightGBM parameters, CatBoost parameter tuning.