Definition
A random forest has more than a dozen settings: how many trees, how deep, how many features per split. Leave them at their defaults and you may be close to the best; choose them badly and accuracy can drop sharply. Tuning finds good settings systematically instead of by guesswork.
Hyperparameters are settings chosen before training rather than learned from data, such as k in k-nearest neighbours, C in an SVM, the depth of a tree or the learning rate of gradient boosting. Hyperparameter tuning searches for the values that give the best validation performance. Grid search tries every combination from lists of values; random search samples combinations at random from ranges; both usually evaluate each candidate with cross-validation. The final model is then evaluated once on an untouched test set.
In simple words
Tuning is like adjusting the knobs on a radio to get the clearest signal. Grid search tries every knob position in fixed steps; random search tries random positions, which often finds a clear signal faster when only one or two knobs really matter.
Overview
- Hyperparameters are settings chosen before training, such as tree depth, k or the learning rate.
- Tuning searches for the values that give the best validation performance.
- Each candidate is usually evaluated with cross-validation.
- Strengths and rates are searched on a logarithmic scale.
- The test set is used only once, after tuning is finished.
- Tuning gains are often small; good defaults and features matter more.
Why it matters
Well-tuned settings can turn a mediocre model into a strong one, and tuning done badly, on the test set, produces scores that will not hold in real use. Knowing how to tune fairly and efficiently is a core practical skill.
Key terms
- Parameter vs hyperparameter
- Parameters (weights, splits) are learned in training; hyperparameters are set before it.
- Grid search
- Evaluating every combination of listed values.
- Random search
- Evaluating a fixed number of randomly sampled combinations.
- Search space
- The ranges and scales of values to try, such as 0.001 to 100 on a log scale.
- GridSearchCV
- Scikit-learn's tool that runs grid search with cross-validation and refits the best model.
- Validation overfitting
- Choosing among so many settings that the best one looks good partly by luck.
Types
There are several types of hyperparameter search, from exhaustive to adaptive. These are the main ones.
1. Manual search
Trying settings by hand, guided by experience and plots.
- How it works: Change one hyperparameter at a time, watch validation curves and keep notes.
- Example: Lowering the learning rate after seeing the training loss jump around.
- When to use it: Use it to build intuition and narrow the ranges for automatic search. Record every trial so results can be compared.
2. Grid search
Evaluating every combination of listed values.
- How it works: GridSearchCV trains k × (product of list sizes) models; cost multiplies with each added hyperparameter.
- Example: 3 depths × 3 leaf sizes × 5 folds = 45 trainings for a random forest.
- When to use it: Use it for two or three hyperparameters with a few values each.
3. Random search
Evaluating a fixed number of randomly sampled combinations.
- How it works: RandomizedSearchCV samples from ranges or distributions; with the same budget it tries many more distinct values per hyperparameter (Bergstra and Bengio, 2012).
- Example: 30 random trials over depth, leaf size and max_features.
- When to use it: Use it as the default for more than two or three hyperparameters.
4. Bayesian optimisation
Choosing the next settings to try based on results so far.
- How it works: A surrogate model predicts which settings are promising and balances exploring new areas with refining good ones; tools include Optuna and scikit-optimize.
- Example: Optuna finding a good gradient-boosting configuration in 50 trials.
- When to use it: Use it when each training run is expensive.
5. Successive halving and early-stopping searches
Giving many configurations a small budget, then more budget only to the best.
- How it works: HalvingGridSearchCV and Hyperband train all candidates briefly, keep the top fraction and repeat with more data or epochs.
- Example: Starting 81 neural-network configurations for one epoch and finishing only the best 3.
- When to use it: Use it for large search spaces and costly models such as neural networks.
Explained in detail
Grid search and random search
Grid search is simple and thorough. To tune a random forest's max_depth over [4, 8, 16, None] and min_samples_leaf over [1, 5, 20], it trains and cross-validates 4 × 3 = 12 combinations; with 5-fold cross-validation that is 60 training runs. The cost multiplies with every hyperparameter added, and many runs are wasted on settings that hardly matter. Random search, studied by James Bergstra and Yoshua Bengio in 2012, samples, say, 30 random combinations from ranges or distributions. Because it tries 30 different values of each hyperparameter instead of a handful, it usually finds settings at least as good with far fewer runs, especially when only a few hyperparameters strongly affect the score.
Common hyperparameters to tune
| Model | Hyperparameters | Typical search space |
|---|---|---|
| k-NN | n_neighbors, weights | k from 1 to 50; uniform or distance |
| Logistic regression | C (inverse regularisation) | 0.001 to 100, log scale |
| SVM (RBF) | C, gamma | Both 0.001 to 1,000, log scale |
| Random forest | max_depth, min_samples_leaf, max_features | Depth 4 to 30; leaf 1 to 20; sqrt or 0.3 to 0.7 |
| Gradient boosting | learning_rate, max_depth, n_estimators, subsample | 0.01 to 0.3; depth 2 to 8; early stopping for trees |
Tuning fairly
The golden rule is that the test set plays no part in tuning. Use cross-validation on the training data, or a separate validation set, to compare settings; GridSearchCV and RandomizedSearchCV do this and then refit the best configuration on all training data. Put preprocessing inside a pipeline so it is refitted on each fold. Search regularisation strengths and learning rates on a logarithmic scale, because the difference between 0.001 and 0.01 matters as much as between 1 and 10. If the best value lies at the edge of your range, widen the range. When many configurations are compared on a small data set, the winner's cross-validation score is optimistic, so the final test-set score is the honest number to report.
Smarter methods exist for expensive models: Bayesian optimisation tools such as Optuna choose the next setting to try based on earlier results, and successive halving gives more training budget only to promising settings. For most intermediate projects, random search with 20 to 50 trials and cross-validation is an excellent default. Tuning does not always pay off: in the code below, grid search picks max_depth 6 with cross-validated accuracy 0.960, yet on the test set the tuned forest scores 0.944, exactly the same as the default forest. Random forests have good defaults, so always compare tuned models with the defaults before claiming an improvement.
How it works
- Split off a test set and do not touch it.
- Build a pipeline of preprocessing and model.
- Define a search space, using log scales for strengths and rates.
- Run GridSearchCV or RandomizedSearchCV with 5-fold cross-validation.
- Inspect results, refit the best model and evaluate it once on the test set.
The maths
- Number of cross-validation folds
- Number of values tried for hyperparameter j
Tuning an SVM with C in [0.1, 1, 10, 100] and gamma in [0.001, 0.01, 0.1, 1] gives 4 × 4 = 16 combinations; with 5-fold cross-validation that is 80 trainings. Adding a third hyperparameter with 5 values would make 400. Suppose the best cross-validation accuracy is 0.91 at C = 10, gamma = 0.01, and its neighbours score 0.90 and 0.89. That stable plateau is reassuring. If instead the best were at C = 100, the edge of the grid, you would extend the grid to 1,000 before concluding.
In code
import numpy as np
from scipy.stats import randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
rf = RandomForestClassifier(n_estimators=150, random_state=0)
grid = GridSearchCV(rf, {"max_depth": [3, 6, None], "min_samples_leaf": [1, 5, 20]}, cv=5).fit(X_tr, y_tr)
print("Grid search:", grid.best_params_, f"CV accuracy={grid.best_score_:.3f}", f"({len(grid.cv_results_['params'])} combinations)")
space = {"max_depth": randint(2, 20), "min_samples_leaf": randint(1, 30), "max_features": ["sqrt", 0.3, 0.6]}
rand = RandomizedSearchCV(rf, space, n_iter=12, cv=5, random_state=0).fit(X_tr, y_tr)
print("Random search:", rand.best_params_, f"CV accuracy={rand.best_score_:.3f}")
for name, search in [("grid", grid), ("random", rand)]:
print(f"Test accuracy of best {name} model: {search.score(X_te, y_te):.3f}")
print("Default forest test accuracy:", round(rf.fit(X_tr, y_tr).score(X_te, y_te), 3))Output
Grid search: {'max_depth': 6, 'min_samples_leaf': 1} CV accuracy=0.960 (9 combinations)
Random search: {'max_depth': 14, 'max_features': 0.6, 'min_samples_leaf': 2} CV accuracy=0.958
Test accuracy of best grid model: 0.944
Test accuracy of best random model: 0.937
Default forest test accuracy: 0.944Try it yourself
Parameter or hyperparameter?
Is each one learned in training or set before it?
The weights of a logistic regression
The maximum depth of a tree
The number of neighbours k
The split thresholds inside a tree
The learning rate of gradient boosting
Fair tuning workflow
Put the steps in order.
- 1Set aside a test set
- 2Define the search space
- 3Refit the best settings and test once
- 4Search with cross-validation on training data
- 5Build a preprocessing-and-model pipeline
Common mistakes
- Tuning on the test set and reporting the best test score.
- Using a linear scale for C or learning rates.
- Running a huge grid when random search would find a good result faster.
Check yourself
Summary
- Hyperparameters are set before training; tuning searches for the best values.
- Grid search is exhaustive; random search is usually more efficient.
- Tune with cross-validation inside a pipeline, use log scales, and test once at the end.
Finished reading?
Sources: Bergstra and Bengio (2012), Random Search for Hyper-Parameter Optimization · scikit-learn user guide