A1 · Supervised Machine Learning in Depth

Hyperparameter Tuning

Hyperparameters are settings chosen before training, such as tree depth, regularisation strength or learning rate. Tuning searches for the values that give the best cross-validated performance. Grid search tries every combination on a grid; random search samples combinations at random and is more efficient when only a few hyperparameters matter; Bayesian optimisation, as in Optuna, models the score to choose promising values next.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

How do you find the best settings without trying every combination?

Hyperparameters are settings chosen before training, such as tree depth, regularisation strength or learning rate. Tuning searches for the values that give the best cross-validated performance. Grid search tries every combination on a grid; random search samples combinations at random and is more efficient when only a few hyperparameters matter; Bayesian optimisation, as in Optuna, models the score to choose promising values next.

In simple words

Grid search is checking every seat in a cinema for your lost phone, row by row. Random search checks seats at random and, surprisingly often, finds it sooner, because what matters is usually one or two settings. Bayesian search remembers where you've looked and where it was 'warmer'.

Why it matters

Good tuning can turn an average model into a strong one, but it is also the easiest place to overfit the validation data. A disciplined search, with cross-validation and a final untouched test set, gives gains you can trust.

Key terms

Hyperparameter
A setting chosen before training, such as tree depth or regularisation strength, rather than learned from data.
Grid search
Trying every combination from a list of values for each hyperparameter.
Random search
Sampling random combinations from ranges or distributions.
Bayesian optimisation
Building a model of score versus settings to choose promising settings to try next.
Successive halving
Starting many configurations with small budgets and giving more resources only to the best.
Search space
The ranges and scales, such as logarithmic for learning rates, over which settings are searched.

Explained in detail

Grid, random and smarter searches

Grid search is exhaustive but wasteful: tuning 4 hyperparameters with 5 values each means 625 combinations, each trained k times in cross-validation. Many of those trials vary hyperparameters that barely matter. James Bergstra and Yoshua Bengio showed in 2012 that random search, sampling settings at random from sensible ranges, usually finds equally good or better settings with far fewer trials, because it explores more distinct values of the hyperparameters that do matter. Bayesian optimisation, implemented in libraries such as Optuna and scikit-optimize, goes further: it models how the score depends on the settings and picks the next trial where improvement is most likely, which is valuable when each training run is expensive.

Designing the search space

HyperparameterSensible rangeScale
Learning rate (boosting)0.01 to 0.3Logarithmic
Regularisation C or alpha0.0001 to 1,000Logarithmic
Tree depth3 to 12Integer, linear
Subsample fraction0.5 to 1.0Linear
Number of treesSet high and use early stoppingNot searched directly

Tune on validation data or with cross-validation on the training set, never on the test set. Start broad and coarse, then narrow around the best region. Successive halving and Hyperband, available as HalvingRandomSearchCV in scikit-learn, start many configurations with a small budget, such as few trees or a fraction of the data, and promote only the best to larger budgets, which saves a great deal of time. Keep a record of every trial; plots of score against each hyperparameter show which ones matter and whether the best value lies at the edge of the range, in which case the range should be widened.

Tuning usually gives modest gains compared with better data and features, and heavy tuning on a small validation set can overfit it. Report the final model's performance on the untouched test set, and record the chosen hyperparameters in the model card so the result can be reproduced.

How it works

MethodHow it choosesBest for
Grid searchEvery combinationTwo or three parameters, small grids
Random searchRandom samples from rangesMany parameters, few that matter
Bayesian optimisationA model of the score guides the next tryExpensive models, limited budget
Successive halvingDrop weak candidates earlyLarge searches

In code

Grid search and random search over the same budget for a random forest, both scored by cross-validation and checked once on a held-out test set. Optuna works the same way on your own machine.

tuning.py
from scipy.stats import randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
forest = RandomForestClassifier(n_estimators=100, random_state=0)

grid = GridSearchCV(forest, {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5, 10]}, cv=4)
grid.fit(X_tr, y_tr)
print("Grid (9 tries):  ", grid.best_params_, f"CV {grid.best_score_:.3f}  test {grid.score(X_te, y_te):.3f}")

space = {"max_depth": randint(2, 20), "min_samples_leaf": randint(1, 20), "max_features": ["sqrt", "log2", None]}
rand = RandomizedSearchCV(forest, space, n_iter=9, cv=4, random_state=0)
rand.fit(X_tr, y_tr)
print("Random (9 tries):", rand.best_params_, f"CV {rand.best_score_:.3f}  test {rand.score(X_te, y_te):.3f}")

Output

Grid (9 tries):   {'max_depth': 6, 'min_samples_leaf': 1} CV 0.965  test 0.944
Random (9 tries): {'max_depth': 14, 'max_features': 'log2', 'min_samples_leaf': 1} CV 0.965  test 0.944

Try it yourself

Which search when?

Choose the search method for each situation.

  • Eight parameters, only a few likely to matter

  • Each training run takes an hour on a GPU

  • Two parameters with three values each

Common mistakes

  • Tuning on the test set.
  • Searching a linear grid for parameters that act on a log scale.
  • Reporting the best cross-validation score as the expected real-world score. It is optimistic.

Check yourself

Question 1

1. Why can random search beat grid search with the same budget?

Question 2

2. Which is a hyperparameter?

Question 3

3. What should the final test set be used for?

Summary

  • Hyperparameters are chosen before training; tuning searches them with cross-validation.
  • Grid for small spaces, random for many parameters, Bayesian for expensive models.
  • Use log scales and keep the test set for one final check.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Bergstra and Bengio, Random Search for Hyper-Parameter Optimization (2012) · Optuna documentation