Definition
Why do these three libraries win so many data competitions?
XGBoost, LightGBM and CatBoost are optimised gradient-boosting libraries. XGBoost adds second-order (Newton) updates and explicit regularisation of tree complexity. LightGBM bins features into histograms and grows trees leaf-wise for speed on large data. CatBoost uses ordered boosting and built-in target statistics for categorical features. scikit-learn's HistGradientBoosting implements the histogram approach.
In simple words
They all do gradient boosting; they differ in engineering. Histograms turn a million unique values into 256 buckets, so finding splits is fast. Built-in handling of missing values and categories saves preprocessing. Early stopping on a validation set picks the number of trees automatically.
Why it matters
For structured business data, a tuned gradient-boosting library is usually the model to beat. Knowing their differences helps you choose quickly and tune the parameters that matter.
Key terms
- XGBoost
- A gradient-boosting library (2014–2016) with regularised trees, second-order gradients and parallel training.
- LightGBM
- Microsoft's library (2017) using histogram binning and leaf-wise growth for speed on large data.
- CatBoost
- Yandex's library (2017) with ordered target statistics for categorical features and symmetric trees.
- Histogram binning
- Grouping each feature's values into, say, 255 bins so split search is much faster.
- Leaf-wise growth
- Splitting the leaf with the largest gain anywhere in the tree, rather than level by level.
- Monotonic constraint
- Forcing a prediction to only increase (or decrease) with a feature, such as risk with age.
Explained in detail
What these libraries add
All three implement gradient-boosted decision trees but add engineering that made boosting fast and accurate enough to dominate tabular machine learning. XGBoost, created by Tianqi Chen and described in a 2016 paper, uses both the gradient and the second derivative of the loss to choose splits, adds L1 and L2 penalties on leaf values, handles missing values by learning a default direction at each split, and parallelises split finding. LightGBM speeds things up further with histogram binning, leaf-wise growth that reaches lower loss with fewer splits, and techniques that sample rows with small gradients and bundle sparse features. CatBoost focuses on categorical data: it encodes categories with target statistics computed in a random order so a row never uses its own label, which avoids a subtle leakage, and builds balanced symmetric trees that predict very quickly.
Choosing and tuning
| Library | Strong when | Key parameters |
|---|---|---|
| XGBoost | General tabular problems; mature ecosystem and GPU support | eta, max_depth, subsample, colsample_bytree, lambda, n_estimators |
| LightGBM | Very large data sets; fastest training | learning_rate, num_leaves, min_data_in_leaf, feature_fraction |
| CatBoost | Many categorical features; good results with default settings | learning_rate, depth, l2_leaf_reg, iterations |
A practical tuning recipe: set a moderate learning rate such as 0.05, a large maximum number of trees and early stopping on a validation set; tune tree size (max_depth around 4 to 8, or num_leaves around 31 to 255 in LightGBM) and minimum leaf size; then add row and column subsampling around 0.7 to 0.9 and some L2 regularisation; finally, lower the learning rate and retrain for a small extra gain. Watch for LightGBM's leaf-wise trees overfitting small data sets: limit num_leaves and increase min_data_in_leaf. All three support monotonic constraints, useful in credit scoring where regulators expect risk to rise with missed payments, and all integrate with SHAP for explaining predictions.
- On typical tabular data with fewer than a few million rows, these libraries usually outperform neural networks while training far faster.
- Differences between the three are often small once tuned; data quality and features matter more than the library.
Typical scenarios include credit risk, fraud detection, insurance pricing, click-through prediction for advertising, demand forecasting with lag features and ranking search results, where LightGBM and XGBoost provide learning-to-rank objectives.
How it works
| Library | Strength | Reach for it when |
|---|---|---|
| XGBoost | Mature, regularised, widely deployed | A solid default with strong tooling |
| LightGBM | Very fast on large data, leaf-wise growth | Millions of rows, many features |
| CatBoost | Native categorical handling, robust defaults | Many high-cardinality categories |
| HistGradientBoosting | In scikit-learn, histogram-based | You want no extra dependency |
The parameters that matter most: learning rate, number of trees (with early stopping), tree size (max_depth or num_leaves), minimum samples per leaf, and subsampling of rows and columns.
The maths
- First and second derivatives of the loss
- Number of leaves
- Leaf weights
In code
Histogram gradient boosting with early stopping and native missing-value handling. The same pattern works with xgboost.XGBClassifier or lightgbm.LGBMClassifier on your own machine.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=6000, n_features=30, n_informative=12, random_state=0)
rng = np.random.default_rng(0)
X[rng.random(X.shape) < 0.05] = np.nan # 5 per cent missing values
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
hgb = HistGradientBoostingClassifier(learning_rate=0.1, max_iter=500, early_stopping=True,
validation_fraction=0.15, n_iter_no_change=20, random_state=0)
hgb.fit(X_tr, y_tr) # missing values are handled natively
print("Trees used after early stopping:", hgb.n_iter_)
print("Histogram boosting test accuracy:", round(hgb.score(X_te, y_te), 3))
X_filled = np.nan_to_num(X_tr, nan=0.0) # classic GradientBoosting needs no NaNs
gb = GradientBoostingClassifier(n_estimators=hgb.n_iter_, random_state=0).fit(X_filled, y_tr)
print("Classic boosting test accuracy:", round(gb.score(np.nan_to_num(X_te, nan=0.0), y_te), 3))Output
Trees used after early stopping: 189 Histogram boosting test accuracy: 0.916 Classic boosting test accuracy: 0.872
Try it yourself
Which library when?
Match each situation to the library you'd try first.
- 50 million rows and speed matters most
- Dozens of categorical columns with thousands of values each
- No extra libraries allowed in production
- A well-tested default with broad deployment support
Common mistakes
- Fixing the number of trees by hand instead of early stopping.
- One-hot encoding hundreds of categories for CatBoost, which handles them natively.
- Tuning everything at once. Start with learning rate, tree size and early stopping.
Check yourself
Summary
- XGBoost, LightGBM and CatBoost are engineered gradient boosting.
- Histograms, missing-value handling and early stopping make them fast and practical.
- Tune learning rate, tree size and subsampling; let early stopping pick the trees.
Finished reading?
Sources: XGBoost documentation · LightGBM documentation · CatBoost documentation