A1 · Supervised Machine Learning in Depth

Stacking and Voting Ensembles

Voting ensembles combine different models by majority vote (hard voting) or by averaging predicted probabilities (soft voting). Stacking trains a meta-model on the out-of-fold predictions of several base models, learning how much to trust each. Both work best when the base models are accurate and make different kinds of errors.

10 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Can two very different experts together beat either one?

Voting ensembles combine different models by majority vote (hard voting) or by averaging predicted probabilities (soft voting). Stacking trains a meta-model on the out-of-fold predictions of several base models, learning how much to trust each. Both work best when the base models are accurate and make different kinds of errors.

In simple words

A linear model, a k-NN model and a forest see the data differently. Where one is wrong, the others are often right. Voting lets them outvote each other's mistakes; stacking hires a manager who learns from experience which expert to listen to in which situation.

Why it matters

Ensembles of diverse models often give the last one or two per cent of accuracy in competitions and critical applications, at the cost of more complexity and slower prediction.

Key terms

Hard voting
Each model votes for a class; the majority wins.
Soft voting
Averaging predicted probabilities and choosing the highest, usually better than hard voting.
Stacking
Training a meta-model on the predictions of several base models.
Meta-learner
The second-level model, often logistic or ridge regression, that combines base predictions.
Out-of-fold predictions
Base-model predictions made on rows not used to train that model, so the meta-learner is not fooled.
Model diversity
Base models that make different kinds of errors, which is what makes combining useful.

Explained in detail

Why combining different models helps

If three classifiers are each 70 per cent accurate and their errors were completely independent, a majority vote would be right about 78 per cent of the time, because it fails only when at least two are wrong together. Real models make correlated errors, so gains are smaller, but combining genuinely different models, such as a logistic regression, a random forest and a gradient-boosting model, often improves accuracy and stability. Voting is the simplest combination. Soft voting averages probabilities, letting a confident model outweigh two uncertain ones; weights can favour stronger models. Averaging the predictions of several neural networks trained with different random seeds is the same idea and is common in competitions and medical imaging.

How stacking is done correctly

Stacking, introduced by David Wolpert in 1992, learns how to combine base models. Each base model is trained with k-fold cross-validation, and its predictions on each held-out fold are collected, so every training row gets a prediction from a model that did not see it. These out-of-fold predictions become the features of the meta-learner, usually a simple, regularised model that learns, for example, to trust the random forest more for some kinds of rows. The base models are then retrained on all training data for use at prediction time. Training the meta-learner on in-sample predictions instead would leak, because base models are overconfident on data they memorised. Scikit-learn's StackingClassifier and StackingRegressor handle this automatically.

MethodEffortTypical gainCost
Hard votingVery lowSmallSeveral models to run
Soft votingLowSmall to moderateNeeds calibrated probabilities
StackingModerateOften the best single gainComplex to maintain; slower predictions

Ensembles have costs: more computation, larger deployments and harder explanations. In production, the extra half-percent of accuracy that wins a competition may not justify running five models. A good rule is to ensemble only when the gain on validation data is meaningful for the decision, and to prefer diversity over quantity: two different strong models usually help more than ten similar ones.

A practical example: a team predicting crop yield combines a gradient-boosting model that uses weather and soil features, a k-NN model that finds similar past farms and a linear model on satellite vegetation indices. Each is right in different situations, and a ridge-regression meta-learner trained on out-of-fold predictions reduces the error noticeably compared with the best single model.

How it works

  1. Choose diverse, individually good base models.
  2. Voting: combine their votes or probabilities directly.
  3. Stacking: generate out-of-fold predictions with cross-validation, so the meta-model never sees predictions on rows the base models trained on.
  4. Train a simple meta-model, often logistic regression, on those predictions.

In code

Three different base models, hard and soft voting, and a stacked ensemble, all cross-validated.

stacking.py
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier, StackingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
base = [
    ("logreg", make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(7))),
    ("forest", RandomForestClassifier(n_estimators=200, random_state=0)),
]
for name, model in base:
    print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.4f}")

ensembles = {
    "hard voting": VotingClassifier(base, voting="hard"),
    "soft voting": VotingClassifier(base, voting="soft"),
    "stacking": StackingClassifier(base, final_estimator=LogisticRegression(max_iter=1000), cv=5),
}
for name, model in ensembles.items():
    print(f"{name:>13}: {cross_val_score(model, X, y, cv=5).mean():.4f}")

Output

       logreg: 0.9807
          knn: 0.9701
       forest: 0.9596
  hard voting: 0.9807
  soft voting: 0.9737
     stacking: 0.9789

Try it yourself

Hard or soft voting?

Three models give P(positive) = 0.9, 0.4 and 0.45. Match each method to its decision.

  • Hard voting (threshold 0.5 per model)
  • Soft voting (average probability)

Common mistakes

  • Stacking on in-sample predictions, which leaks and overfits the meta-model.
  • Combining near-identical models, which adds cost but no diversity.
  • Using soft voting with models whose probabilities are badly calibrated.

Check yourself

Question 1

1. Soft voting averages…

Question 2

2. Why must stacking use out-of-fold predictions?

Question 3

3. Ensembles help most when base models…

Summary

  • Voting combines votes (hard) or probabilities (soft).
  • Stacking learns a meta-model on out-of-fold predictions.
  • Diverse, accurate base models make ensembles work.

Finished reading?

Sources: scikit-learn: Stacking · Wolpert, Stacked Generalization (1992)