Definition
Your linear model beats persistence. Now the temptation is to try every algorithm on the internet. A disciplined approach, one change at a time, judged on validation data and finally on an untouched test year, separates real progress from lucky noise.
Improving a model means making targeted changes, such as better features, more or better data, a more flexible algorithm or tuned hyperparameters, and keeping only those that improve performance on validation data. Evaluating it properly means testing once on held-out data, reporting several metrics that match the users' goals, examining errors by condition and group, checking uncertainty, and comparing against the baseline. For a decision such as sending alerts, it also means choosing a threshold that balances missed dangerous days against false alarms.
In simple words
A cricketer improving their batting changes one thing at a time, their stance, their grip, their footwork, and checks each change in practice matches before the real tournament. Model improvement works the same way.
Overview
- After a baseline, improve the model step by step and measure each change.
- Improvements come from better data, better features, better models and better tuning.
- Use cross-validation or a validation set for choices; keep the test set for the final check.
- Choose metrics that match the problem, such as recall for screening or MAE for forecasts.
- Error analysis, looking at wrong predictions, often shows the next improvement.
- Keep a log of experiments with settings and scores.
Why it matters
Uncontrolled tinkering produces models that look good on paper and fail in use. A careful loop gives honest results, explains which changes mattered and shows where the model is still weak.
Key terms
- Hyperparameter tuning
- Searching for good settings, such as tree depth, using validation data.
- Ablation
- Removing one feature or component to see how much it contributes.
- Error analysis
- Studying where and why the model makes its biggest mistakes.
- Decision threshold
- The cut-off that turns a prediction into an action, such as alert if forecast > 90.
- Precision-recall trade-off
- Lowering the threshold catches more bad days but raises false alarms.
- Confidence interval
- A range showing how much a metric might vary, for example from bootstrapping the test set.
Types
There are several types of improvement and evaluation method. These are the main ones for a project.
1. Feature engineering
Creating more informative inputs from the raw data.
- How it works: Add ratios, date parts, rolling averages, counts or text features, and remove noisy or leaking features.
- Example: Adding "festival week" and "last week's sales" features to a sales forecast.
- When to use it: Use it first: good features often help more than a fancier model.
2. Trying stronger models
Comparing several model families on the same validation data.
- How it works: Try random forests, gradient boosting, SVMs or neural networks and compare with the baseline using cross-validation.
- Example: Gradient boosting reducing MAE from 14 to 9 units compared with linear regression.
- When to use it: Use it after features are solid, choosing the simplest model that is clearly better.
3. Hyperparameter tuning
Searching for the best settings of a chosen model.
- How it works: Use GridSearchCV or RandomizedSearchCV with cross-validation, searching only on training data.
- Example: Finding that max_depth=6 and 300 trees work best for a random forest.
- When to use it: Use it near the end; gains are usually smaller than from data and features.
4. Error analysis
Studying the examples the model gets wrong.
- How it works: Sort errors by size or type, look for patterns such as one group, one season or noisy labels, and fix the cause.
- Example: Finding most errors come from new shops with little history, and adding a feature for shop age.
- When to use it: Use it after every major experiment to guide the next step.
5. Final evaluation and fairness checks
Testing the chosen model once on untouched test data and across groups.
- How it works: Report the main metric, a confusion matrix or error plot, confidence intervals and per-group results; never tune on the test set.
- Example: Reporting 87% recall overall, 89% for urban and 82% for rural users.
- When to use it: Use it once at the end to give an honest estimate of real-world performance.
Explained in detail
A disciplined improvement loop
Start from the error analysis of the baseline model. In our project the biggest errors come on days when pollution jumps suddenly, such as calm, cold nights after windy days, and around festivals. That suggests specific changes: add the change in wind from today to tomorrow using the weather forecast, add a festival-day indicator and try a model that can capture interactions, such as a random forest or gradient-boosted trees. Make one change, train on the training period, measure on the validation period, and record the result in the experiment table. Keep a change only if it helps on validation data; otherwise revert it. Ablation tests then show which features matter most, which helps explain the model to users.
Evaluating for the real decision
| Question | How to check |
|---|---|
| Is it better than the baseline? | MAE and RMSE on the untouched test year, with a confidence interval |
| Does it catch dangerous days? | Recall and precision for Poor-or-worse days at the chosen threshold |
| Where does it fail? | Errors by season, by pollution level and on festival days |
| Is it fair across places? | Errors by monitoring station or area of the city |
| Is it stable? | Results across several years or rolling test periods |
Choosing the alert threshold is a decision about values, not just accuracy. If alerts go out whenever the forecast exceeds 90, the model may miss days that turn out slightly above 90; lowering the threshold to 80 catches more bad days at the cost of more false alarms. Because principals told us that a missed bad day is worse than an extra alert, we choose the lowest threshold that keeps false alarms at a level they find acceptable, perhaps one unnecessary alert in four. Finally, run the chosen model once on the test year, report all metrics honestly, including where it is weak, and compare with persistence. A small but consistent gain that users value is a real success.
Keep the experiment table up to date: date, change made, features, model, settings, validation scores and a one-line conclusion. It becomes the backbone of your report and presentation, and it lets a teammate or examiner retrace every decision. When two versions score almost the same, prefer the simpler one, because it is easier to explain, cheaper to run and less likely to break.
How it works
Follow the first month of predictions
Save each forecast with its timestamp, model version, units and whether a sensor was missing. Compare training and live input distributions. A jump in missing readings can expose a broken feed before tomorrow's true pollution level arrives. A change in the input distribution is a reason to investigate. By itself it cannot tell you how much the model's error changed.
Once observations arrive, align each with the forecast made for that date. Compare errors with the persistence baseline on the same dates and by station. Test that training and serving convert units in the same way. If a feed silently switches units, fix the conversion and replay saved cases. Retraining on the broken feed would teach the model the fault.
- Study the baseline's biggest errors and form hypotheses.
- Make one change at a time and measure on validation data.
- Keep helpful changes, revert others and run ablations.
- Choose the alert threshold with users, balancing recall and false alarms.
- Test once on the untouched test period and report all metrics and weaknesses.
In code
Adding tomorrow's wind forecast and the festival flag gives the biggest gain. The random forest is not better than the linear model here, a useful reminder that a more complex algorithm does not always win; the validation data, not the algorithm's reputation, decides.
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
def make_city_data(days=3 * 365, seed=3):
rng = np.random.default_rng(seed)
d = pd.date_range("2022-01-01", periods=days, freq="D")
winter = np.cos((d.dayofyear - 15) / 365 * 2 * np.pi) # +1 in mid-January, -1 in mid-July
# For each date, generate the forecast issued the previous day, then the realised wind.
forecast = np.clip(rng.normal(3 - winter, 1.0), 0.3, None)
wind = np.clip(forecast + rng.normal(0, 0.7, days), 0.3, None)
humidity = np.clip(rng.normal(60 + 15 * winter, 10), 20, 100)
festival = ((d.month == 11) & (d.day <= 3)).astype(int)
pm = np.zeros(days); pm[0] = 120
for t in range(1, days):
level = 85 + 60 * winter[t] - 14 * (wind[t] - 3) + 0.3 * (humidity[t] - 60) + 150 * festival[t]
pm[t] = 0.45 * pm[t - 1] + 0.55 * level + rng.normal(0, 10)
return pd.DataFrame({"pm25": np.clip(pm, 10, None), "wind": wind, "prior_forecast": forecast, "humidity": humidity, "festival": festival}, index=d)
df = make_city_data()
df["target"] = df["pm25"].shift(-1)
df["pm25_3day"] = df["pm25"].rolling(3).mean()
df["season"] = np.cos((df.index.dayofyear - 15) / 365 * 2 * np.pi)
df["wind_tomorrow_fc"] = df["prior_forecast"].shift(-1) # already issued today for tomorrow
df["festival_tomorrow"] = df["festival"].shift(-1)
df = df.dropna()
label_date = df.index + pd.Timedelta(days=1)
train = df[label_date < "2023-07-01"]
valid = df[(df.index >= "2023-07-01") & (label_date < "2024-01-01")]
test = df[df.index >= "2024-01-01"]
basic = ["pm25", "pm25_3day", "wind", "humidity", "season"]
richer = basic + ["wind_tomorrow_fc", "festival_tomorrow"]
mae = lambda pred, part: np.mean(np.abs(pred - part["target"]))
print(f"Validation MAE persistence: {mae(valid['pm25'].values, valid):.1f}")
candidates = [
("linear, basic", LinearRegression(), basic),
("linear, forecast and festival", LinearRegression(), richer),
("forest, forecast and festival", RandomForestRegressor(200, min_samples_leaf=5, random_state=0), richer),
]
results = []
for name, model, features in candidates:
model.fit(train[features], train["target"])
predictions = model.predict(valid[features])
error = mae(predictions, valid)
results.append((error, name, model, features, predictions))
print(f"Validation MAE {name}: {error:.1f}")
_, name, chosen, features, predictions = min(results, key=lambda result: result[0])
# Classroom cost assumptions. One missed high-pollution day costs five false alarms.
def alert_cost(cutoff):
alerts = predictions > cutoff
high = valid["target"] > 90
return 5 * (high & ~alerts).sum() + (~high & alerts).sum()
threshold = min([90, 80, 70], key=alert_cost)
print(f"Selected on validation: {name}, threshold {threshold}")
# Freeze the model and decision rule before inspecting the test year.
pred = chosen.predict(test[features])
print(f"Test MAE model: {mae(pred, test):.1f} persistence: {mae(test['pm25'].values, test):.1f}")
bad = test["target"] > 90
alert = pred > threshold
print(f"Test recall: {(alert & bad).sum() / bad.sum():.2f} false alarms: {int((alert & ~bad).sum())}")Output
Validation MAE persistence: 11.4 Validation MAE linear, basic: 10.6 Validation MAE linear, forecast and festival: 8.1 Validation MAE forest, forecast and festival: 9.7 Selected on validation: linear, forecast and festival, threshold 80 Test MAE model: 8.8 persistence: 11.7 Test recall: 0.99 false alarms: 24
Try it yourself
Choosing the alert threshold with principals
In this separate fictional validation exercise, five principals compare alert thresholds for a previous winter.
Good practice or bad practice?
Judge each step in improving the model.
Reporting only the metric that looks best
Trying 50 models on the test set and reporting the best
Changing one feature at a time and checking validation MAE
Checking errors separately for festival days
Common mistakes
- Making several changes at once.
- Tuning on the test set.
- Choosing a threshold without asking the people who act on it.
Check yourself
Summary
- Improve with targeted, one-at-a-time changes judged on validation data.
- Evaluate once on untouched test data with several metrics, error analysis and comparison to the baseline.
- Choose decision thresholds with users, balancing missed events and false alarms.
Finished reading?
Sources: Google ML Crash Course, monitoring pipelines · scikit-learn: model evaluation · Google: Rules of Machine Learning