Definition
A student takes a practice test, studies only the questions they got wrong, retakes it, again studies only the new mistakes, and repeats. Each round fixes what is left. Gradient boosting builds a model the same way, one small tree at a time.
Gradient boosting is an ensemble method that builds models sequentially. It starts with a simple prediction, such as the average target, then repeatedly fits a small decision tree to the current errors (more precisely, to the negative gradient of the loss, which for squared error equals the residuals) and adds that tree's predictions, scaled by a learning rate, to the model. After hundreds of rounds, the sum of many weak trees forms a strong model. Popular implementations include scikit-learn's HistGradientBoosting, XGBoost, LightGBM and CatBoost.
In simple words
Imagine guessing a friend's height. You first guess the class average. They say you are 10 cm too low, so you add a bit. Then 4 cm too low, so you add a little more. Each correction is small, and the guesses home in. Gradient boosting adds many such small corrections, each learned by a tiny tree.
Overview
- Gradient boosting builds an ensemble of small trees one at a time, each correcting the errors left by the previous ones.
- Each new tree is fitted to the negative gradient of the loss, which for squared error is the residuals.
- A learning rate scales each tree's contribution; smaller rates need more trees but generalise better.
- Too many trees overfit, so early stopping on validation data is standard.
- Jerome Friedman described gradient boosting machines in 2001.
- It is usually the most accurate method for tabular data.
Why it matters
Gradient-boosted trees are usually the most accurate method for tabular data such as credit records, sales, sensor logs and medical measurements, and they win many data-science competitions. Understanding how they work helps you tune them and avoid overfitting.
Key terms
- Boosting
- Combining weak models sequentially, each correcting the previous ones.
- Weak learner
- A simple model such as a shallow tree that is only somewhat better than guessing.
- Residual
- The difference between the true value and the current prediction.
- Learning rate (shrinkage)
- The fraction of each new tree's prediction added, such as 0.1.
- Number of estimators
- How many trees are added; too many can overfit.
- Early stopping
- Stopping when validation error stops improving.
Types
Gradient boosting has several well-known implementations and variants. These are the main types.
1. AdaBoost
The original boosting algorithm (Freund and Schapire, 1995), which reweights examples.
- How it works: After each weak learner, misclassified examples get more weight, so the next learner focuses on them; learners are combined with weights based on accuracy.
- Example: The Viola-Jones face detector (2001) used AdaBoost to combine thousands of simple features.
- When to use it: Use it to understand boosting, or with very simple learners such as one-split trees.
2. Classic gradient boosting (GBM)
Friedman's method of fitting trees to gradients of any differentiable loss.
- How it works: GradientBoostingRegressor and GradientBoostingClassifier in scikit-learn; subsample < 1 gives stochastic gradient boosting.
- Example: 48 trees chosen by early stopping gave a test error of 11.0 in this module's example.
- When to use it: Use it for small to medium tabular data when you want a well-understood method.
3. Histogram-based boosting (LightGBM, HistGradientBoosting)
Fast boosting that bins feature values into histograms before finding splits.
- How it works: Binning into about 256 buckets makes split search much faster; LightGBM (Microsoft, 2017) also grows trees leaf-wise.
- Example: Training on 10 million rows of click data in minutes.
- When to use it: Use it for large data sets; scikit-learn's HistGradientBoosting handles missing values natively.
4. XGBoost
An optimised, regularised gradient-boosting library by Tianqi Chen and Carlos Guestrin (2016).
- How it works: It adds L1 and L2 penalties on leaf weights, uses second-order gradients, handles sparse data and runs in parallel.
- Example: Widely used in Kaggle-winning solutions and in bank credit-risk models.
- When to use it: Use it when you need top accuracy and fine control over regularisation.
5. CatBoost
A gradient-boosting library by Yandex (2017) designed for categorical features.
- How it works: It uses ordered target statistics to encode categories without leakage and builds symmetric trees that predict quickly.
- Example: Predicting delivery times with features such as city, restaurant and cuisine without manual encoding.
- When to use it: Use it when data has many categorical columns, often with good default settings.
Explained in detail
How boosting builds a model step by step
Gradient boosting for regression follows a simple loop. Start with F₀, the average target. Compute the residuals, the true values minus the current predictions. Fit a small tree, often of depth 3 to 6, that predicts those residuals from the features. Add the tree's prediction multiplied by a learning rate ν, such as 0.1, to get the new model F₁ = F₀ + ν·tree₁. Repeat. Each tree is weak on its own, but together they model complex, non-linear patterns and interactions. The name comes from the fact that the residuals are the negative gradient of the squared-error loss, so each step moves the predictions downhill on the loss, like gradient descent; other losses, such as log loss for classification, use their own gradients.
Bagging versus boosting
| Random forest (bagging) | Gradient boosting | |
|---|---|---|
| Trees | Deep, trained in parallel | Shallow, trained in sequence |
| Reduces mainly | Variance | Bias |
| More trees | Never hurts accuracy | Can overfit; use early stopping |
| Tuning effort | Low | Moderate: learning rate, depth, number of trees |
| Typical accuracy on tabular data | Very good | Often the best |
Tuning without overfitting
The learning rate and the number of trees work together: a smaller learning rate such as 0.05 needs more trees but usually generalises better. The safest recipe is to set a modest learning rate, allow many trees and use early stopping, which watches a validation set and stops adding trees when its error has not improved for, say, 20 rounds. Keep trees shallow, typically depth 3 to 6, and add row subsampling (for example 0.8 of rows per tree), which makes stochastic gradient boosting more robust. Because each tree chases the remaining errors, boosting can overfit noisy or mislabelled data more than a random forest, so check validation curves carefully. In the code below, training error keeps falling as trees are added, from 17.0 with one tree to 1.3 with 500, but validation error is lowest at 48 trees (about 11.8) and then rises to 13.3: after that point, extra trees only memorise noise. Early stopping keeps the 48 trees chosen on validation data, and the test error is 11.0, slightly better than a random forest's 11.4.
Libraries such as XGBoost (2016), LightGBM (2017) and CatBoost (2017) made gradient boosting much faster and added regularisation, built-in handling of missing values and, in CatBoost, careful treatment of categorical features. They are the standard tools for tabular problems in industry, from fraud detection at banks to demand forecasting for retailers.
How it works
- Hold out a validation set (or use cross-validation).
- Start with learning rate 0.1, shallow trees and early stopping.
- Tune depth and the minimum leaf size; add row subsampling.
- Lower the learning rate and allow more trees for a final gain.
- Compare with a random forest and check errors on the test set once.
The maths
- The tree fitted to the current residuals
- Learning rate, for example 0.1
Three flats cost 50, 60 and 90 lakh. Round 0 predicts the mean, 66.7 lakh, for all three, so the residuals are −16.7, −6.7 and +23.3. A one-split tree groups the first two flats (average residual −11.7) and the third (+23.3). With learning rate 0.5 the new predictions are 66.7 − 5.8 = 60.8, 60.8 and 66.7 + 11.7 = 78.3, leaving residuals of −10.8, −0.8 and +11.7. The total squared error falls from 278 + 44 + 544 = 866 to 117 + 1 + 136 = 254. The next tree attacks what remains.
In code
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.model_selection import train_test_split
# The worked example by hand: one round with learning rate 0.5
y = np.array([50.0, 60.0, 90.0])
pred = np.full(3, y.mean())
resid = y - pred
tree_out = np.array([resid[:2].mean(), resid[:2].mean(), resid[2]])
pred = pred + 0.5 * tree_out
print("After one round:", pred.round(1), " squared error:", round(((y - pred) ** 2).sum()))
# Noisy, non-linear data split into training, validation and test sets
rng = np.random.default_rng(0)
X = rng.uniform(-3, 3, size=(600, 5))
t = 20 * np.sin(X[:, 0]) + 10 * X[:, 1] * (X[:, 2] > 0) + rng.normal(0, 12, 600)
X_rest, X_te, t_rest, t_te = train_test_split(X, t, test_size=0.2, random_state=0)
X_tr, X_val, t_tr, t_val = train_test_split(X_rest, t_rest, test_size=0.25, random_state=0)
gb = GradientBoostingRegressor(n_estimators=500, learning_rate=0.1, max_depth=3, subsample=0.8, random_state=0).fit(X_tr, t_tr)
train_err = [np.mean(np.abs(p - t_tr)) for p in gb.staged_predict(X_tr)]
val_err = [np.mean(np.abs(p - t_val)) for p in gb.staged_predict(X_val)]
for rounds in [1, 20, 50, 200, 500]:
print(f"{rounds:>3} trees: train MAE {train_err[rounds - 1]:5.1f} validation MAE {val_err[rounds - 1]:5.1f}")
best = int(np.argmin(val_err)) + 1
test_pred = list(gb.staged_predict(X_te))[best - 1]
print(f"Early stopping keeps {best} trees; test MAE = {np.mean(np.abs(test_pred - t_te)):.1f}")
rf = RandomForestRegressor(300, random_state=0).fit(X_tr, t_tr)
print(f"Random forest test MAE: {np.mean(np.abs(rf.predict(X_te) - t_te)):.1f}")Output
After one round: [60.8 60.8 78.3] squared error: 254 1 trees: train MAE 17.0 validation MAE 18.0 20 trees: train MAE 9.5 validation MAE 12.3 50 trees: train MAE 7.5 validation MAE 11.8 200 trees: train MAE 3.9 validation MAE 12.6 500 trees: train MAE 1.3 validation MAE 13.3 Early stopping keeps 48 trees; test MAE = 11.0 Random forest test MAE: 11.4
Try it yourself
One cycle of gradient boosting
Put the steps in order.
- 1Check validation error and repeat or stop
- 2Fit a small tree to the residuals
- 3Start with a simple prediction, such as the mean
- 4Compute the residuals
- 5Add the tree's predictions times the learning rate
Match the setting to its effect
What happens when you change each?
- Lower learning rate
- Deeper trees
- Early stopping
- Row subsampling 0.8
Common mistakes
- Training thousands of trees with no early stopping.
- Using deep trees in boosting, which quickly overfits.
- Choosing the number of trees on the test set.
Check yourself
Summary
- Gradient boosting adds small trees one at a time, each fitted to the current errors.
- The learning rate, tree depth and number of trees control the fit; use early stopping.
- It is usually the most accurate method for tabular data; XGBoost, LightGBM and CatBoost are popular libraries.
Finished reading?
Sources: Chen and Guestrin (2016), XGBoost: A Scalable Tree Boosting System · Hastie, Tibshirani and Friedman, The Elements of Statistical Learning