Definition
How do you find the best settings without trying every combination?
Hyperparameters are settings chosen before training, such as tree depth, regularisation strength or learning rate. Tuning searches for the values that give the best cross-validated performance. Grid search tries every combination on a grid; random search samples combinations at random and is more efficient when only a few hyperparameters matter; Bayesian optimisation, as in Optuna, models the score to choose promising values next.
In simple words
Grid search is checking every seat in a cinema for your lost phone, row by row. Random search checks seats at random and, surprisingly often, finds it sooner, because what matters is usually one or two settings. Bayesian search remembers where you've looked and where it was 'warmer'.
Why it matters
Good tuning can turn an average model into a strong one, but it is also the easiest place to overfit the validation data. A disciplined search, with cross-validation and a final untouched test set, gives gains you can trust.
Key terms
- Hyperparameter
- A setting chosen before training, such as tree depth or regularisation strength, rather than learned from data.
- Grid search
- Trying every combination from a list of values for each hyperparameter.
- Random search
- Sampling random combinations from ranges or distributions.
- Bayesian optimisation
- Building a model of score versus settings to choose promising settings to try next.
- Successive halving
- Starting many configurations with small budgets and giving more resources only to the best.
- Search space
- The ranges and scales, such as logarithmic for learning rates, over which settings are searched.
Explained in detail
Grid, random and smarter searches
Grid search is exhaustive but wasteful: tuning 4 hyperparameters with 5 values each means 625 combinations, each trained k times in cross-validation. Many of those trials vary hyperparameters that barely matter. James Bergstra and Yoshua Bengio showed in 2012 that random search, sampling settings at random from sensible ranges, usually finds equally good or better settings with far fewer trials, because it explores more distinct values of the hyperparameters that do matter. Bayesian optimisation, implemented in libraries such as Optuna and scikit-optimize, goes further: it models how the score depends on the settings and picks the next trial where improvement is most likely, which is valuable when each training run is expensive.
Designing the search space
| Hyperparameter | Sensible range | Scale |
|---|---|---|
| Learning rate (boosting) | 0.01 to 0.3 | Logarithmic |
| Regularisation C or alpha | 0.0001 to 1,000 | Logarithmic |
| Tree depth | 3 to 12 | Integer, linear |
| Subsample fraction | 0.5 to 1.0 | Linear |
| Number of trees | Set high and use early stopping | Not searched directly |
Tune on validation data or with cross-validation on the training set, never on the test set. Start broad and coarse, then narrow around the best region. Successive halving and Hyperband, available as HalvingRandomSearchCV in scikit-learn, start many configurations with a small budget, such as few trees or a fraction of the data, and promote only the best to larger budgets, which saves a great deal of time. Keep a record of every trial; plots of score against each hyperparameter show which ones matter and whether the best value lies at the edge of the range, in which case the range should be widened.
Tuning usually gives modest gains compared with better data and features, and heavy tuning on a small validation set can overfit it. Report the final model's performance on the untouched test set, and record the chosen hyperparameters in the model card so the result can be reproduced.
How it works
| Method | How it chooses | Best for |
|---|---|---|
| Grid search | Every combination | Two or three parameters, small grids |
| Random search | Random samples from ranges | Many parameters, few that matter |
| Bayesian optimisation | A model of the score guides the next try | Expensive models, limited budget |
| Successive halving | Drop weak candidates early | Large searches |
In code
Grid search and random search over the same budget for a random forest, both scored by cross-validation and checked once on a held-out test set. Optuna works the same way on your own machine.
from scipy.stats import randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
forest = RandomForestClassifier(n_estimators=100, random_state=0)
grid = GridSearchCV(forest, {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5, 10]}, cv=4)
grid.fit(X_tr, y_tr)
print("Grid (9 tries): ", grid.best_params_, f"CV {grid.best_score_:.3f} test {grid.score(X_te, y_te):.3f}")
space = {"max_depth": randint(2, 20), "min_samples_leaf": randint(1, 20), "max_features": ["sqrt", "log2", None]}
rand = RandomizedSearchCV(forest, space, n_iter=9, cv=4, random_state=0)
rand.fit(X_tr, y_tr)
print("Random (9 tries):", rand.best_params_, f"CV {rand.best_score_:.3f} test {rand.score(X_te, y_te):.3f}")Output
Grid (9 tries): {'max_depth': 6, 'min_samples_leaf': 1} CV 0.965 test 0.944
Random (9 tries): {'max_depth': 14, 'max_features': 'log2', 'min_samples_leaf': 1} CV 0.965 test 0.944Try it yourself
Which search when?
Choose the search method for each situation.
Eight parameters, only a few likely to matter
Each training run takes an hour on a GPU
Two parameters with three values each
Common mistakes
- Tuning on the test set.
- Searching a linear grid for parameters that act on a log scale.
- Reporting the best cross-validation score as the expected real-world score. It is optimistic.
Check yourself
Summary
- Hyperparameters are chosen before training; tuning searches them with cross-validation.
- Grid for small spaces, random for many parameters, Bayesian for expensive models.
- Use log scales and keep the test set for one final check.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: Bergstra and Bengio, Random Search for Hyper-Parameter Optimization (2012) · Optuna documentation