I11 · Intermediate Capstone: An End-to-End AI Project

Improving and Evaluating the Model

Improving a model means making targeted changes, such as better features, more or better data, a more flexible algorithm or tuned hyperparameters, and keeping only those that improve performance on validation data. Evaluating it properly means testing once on held-out data, reporting several metrics that match the users' goals, examining errors by condition and group, checking uncertainty, and comparing against the baseline. For a decision such as sending alerts, it also means choosing a threshold that balances missed dangerous days against false alarms.

30 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Your linear model beats persistence. Now the temptation is to try every algorithm on the internet. A disciplined approach, one change at a time, judged on validation data and finally on an untouched test year, separates real progress from lucky noise.

Improving a model means making targeted changes, such as better features, more or better data, a more flexible algorithm or tuned hyperparameters, and keeping only those that improve performance on validation data. Evaluating it properly means testing once on held-out data, reporting several metrics that match the users' goals, examining errors by condition and group, checking uncertainty, and comparing against the baseline. For a decision such as sending alerts, it also means choosing a threshold that balances missed dangerous days against false alarms.

In simple words

A cricketer improving their batting changes one thing at a time, their stance, their grip, their footwork, and checks each change in practice matches before the real tournament. Model improvement works the same way.

Overview

  • After a baseline, improve the model step by step and measure each change.
  • Improvements come from better data, better features, better models and better tuning.
  • Use cross-validation or a validation set for choices; keep the test set for the final check.
  • Choose metrics that match the problem, such as recall for screening or MAE for forecasts.
  • Error analysis, looking at wrong predictions, often shows the next improvement.
  • Keep a log of experiments with settings and scores.

Why it matters

Uncontrolled tinkering produces models that look good on paper and fail in use. A careful loop gives honest results, explains which changes mattered and shows where the model is still weak.

Key terms

Hyperparameter tuning
Searching for good settings, such as tree depth, using validation data.
Ablation
Removing one feature or component to see how much it contributes.
Error analysis
Studying where and why the model makes its biggest mistakes.
Decision threshold
The cut-off that turns a prediction into an action, such as alert if forecast > 90.
Precision-recall trade-off
Lowering the threshold catches more bad days but raises false alarms.
Confidence interval
A range showing how much a metric might vary, for example from bootstrapping the test set.

Types

There are several types of improvement and evaluation method. These are the main ones for a project.

1. Feature engineering

Creating more informative inputs from the raw data.

  • How it works: Add ratios, date parts, rolling averages, counts or text features, and remove noisy or leaking features.
  • Example: Adding "festival week" and "last week's sales" features to a sales forecast.
  • When to use it: Use it first: good features often help more than a fancier model.

2. Trying stronger models

Comparing several model families on the same validation data.

  • How it works: Try random forests, gradient boosting, SVMs or neural networks and compare with the baseline using cross-validation.
  • Example: Gradient boosting reducing MAE from 14 to 9 units compared with linear regression.
  • When to use it: Use it after features are solid, choosing the simplest model that is clearly better.

3. Hyperparameter tuning

Searching for the best settings of a chosen model.

  • How it works: Use GridSearchCV or RandomizedSearchCV with cross-validation, searching only on training data.
  • Example: Finding that max_depth=6 and 300 trees work best for a random forest.
  • When to use it: Use it near the end; gains are usually smaller than from data and features.

4. Error analysis

Studying the examples the model gets wrong.

  • How it works: Sort errors by size or type, look for patterns such as one group, one season or noisy labels, and fix the cause.
  • Example: Finding most errors come from new shops with little history, and adding a feature for shop age.
  • When to use it: Use it after every major experiment to guide the next step.

5. Final evaluation and fairness checks

Testing the chosen model once on untouched test data and across groups.

  • How it works: Report the main metric, a confusion matrix or error plot, confidence intervals and per-group results; never tune on the test set.
  • Example: Reporting 87% recall overall, 89% for urban and 82% for rural users.
  • When to use it: Use it once at the end to give an honest estimate of real-world performance.

Explained in detail

A disciplined improvement loop

Start from the error analysis of the baseline model. In our project the biggest errors come on days when pollution jumps suddenly, such as calm, cold nights after windy days, and around festivals. That suggests specific changes: add the change in wind from today to tomorrow using the weather forecast, add a festival-day indicator and try a model that can capture interactions, such as a random forest or gradient-boosted trees. Make one change, train on the training period, measure on the validation period, and record the result in the experiment table. Keep a change only if it helps on validation data; otherwise revert it. Ablation tests then show which features matter most, which helps explain the model to users.

Evaluating for the real decision

QuestionHow to check
Is it better than the baseline?MAE and RMSE on the untouched test year, with a confidence interval
Does it catch dangerous days?Recall and precision for Poor-or-worse days at the chosen threshold
Where does it fail?Errors by season, by pollution level and on festival days
Is it fair across places?Errors by monitoring station or area of the city
Is it stable?Results across several years or rolling test periods

Choosing the alert threshold is a decision about values, not just accuracy. If alerts go out whenever the forecast exceeds 90, the model may miss days that turn out slightly above 90; lowering the threshold to 80 catches more bad days at the cost of more false alarms. Because principals told us that a missed bad day is worse than an extra alert, we choose the lowest threshold that keeps false alarms at a level they find acceptable, perhaps one unnecessary alert in four. Finally, run the chosen model once on the test year, report all metrics honestly, including where it is weak, and compare with persistence. A small but consistent gain that users value is a real success.

Keep the experiment table up to date: date, change made, features, model, settings, validation scores and a one-line conclusion. It becomes the backbone of your report and presentation, and it lets a teammate or examiner retrace every decision. When two versions score almost the same, prefer the simpler one, because it is easier to explain, cheaper to run and less likely to break.

How it works

Follow the first month of predictions

Save each forecast with its timestamp, model version, units and whether a sensor was missing. Compare training and live input distributions. A jump in missing readings can expose a broken feed before tomorrow's true pollution level arrives. A change in the input distribution is a reason to investigate. By itself it cannot tell you how much the model's error changed.

Once observations arrive, align each with the forecast made for that date. Compare errors with the persistence baseline on the same dates and by station. Test that training and serving convert units in the same way. If a feed silently switches units, fix the conversion and replay saved cases. Retraining on the broken feed would teach the model the fault.

  1. Study the baseline's biggest errors and form hypotheses.
  2. Make one change at a time and measure on validation data.
  3. Keep helpful changes, revert others and run ablations.
  4. Choose the alert threshold with users, balancing recall and false alarms.
  5. Test once on the untouched test period and report all metrics and weaknesses.

In code

Adding tomorrow's wind forecast and the festival flag gives the biggest gain. The random forest is not better than the linear model here, a useful reminder that a more complex algorithm does not always win; the validation data, not the algorithm's reputation, decides.

improve_evaluate.py
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression

def make_city_data(days=3 * 365, seed=3):
    rng = np.random.default_rng(seed)
    d = pd.date_range("2022-01-01", periods=days, freq="D")
    winter = np.cos((d.dayofyear - 15) / 365 * 2 * np.pi)            # +1 in mid-January, -1 in mid-July
    # For each date, generate the forecast issued the previous day, then the realised wind.
    forecast = np.clip(rng.normal(3 - winter, 1.0), 0.3, None)
    wind = np.clip(forecast + rng.normal(0, 0.7, days), 0.3, None)
    humidity = np.clip(rng.normal(60 + 15 * winter, 10), 20, 100)
    festival = ((d.month == 11) & (d.day <= 3)).astype(int)
    pm = np.zeros(days); pm[0] = 120
    for t in range(1, days):
        level = 85 + 60 * winter[t] - 14 * (wind[t] - 3) + 0.3 * (humidity[t] - 60) + 150 * festival[t]
        pm[t] = 0.45 * pm[t - 1] + 0.55 * level + rng.normal(0, 10)
    return pd.DataFrame({"pm25": np.clip(pm, 10, None), "wind": wind, "prior_forecast": forecast, "humidity": humidity, "festival": festival}, index=d)

df = make_city_data()
df["target"] = df["pm25"].shift(-1)
df["pm25_3day"] = df["pm25"].rolling(3).mean()
df["season"] = np.cos((df.index.dayofyear - 15) / 365 * 2 * np.pi)
df["wind_tomorrow_fc"] = df["prior_forecast"].shift(-1)  # already issued today for tomorrow
df["festival_tomorrow"] = df["festival"].shift(-1)
df = df.dropna()
label_date = df.index + pd.Timedelta(days=1)
train = df[label_date < "2023-07-01"]
valid = df[(df.index >= "2023-07-01") & (label_date < "2024-01-01")]
test = df[df.index >= "2024-01-01"]

basic = ["pm25", "pm25_3day", "wind", "humidity", "season"]
richer = basic + ["wind_tomorrow_fc", "festival_tomorrow"]
mae = lambda pred, part: np.mean(np.abs(pred - part["target"]))
print(f"Validation MAE  persistence: {mae(valid['pm25'].values, valid):.1f}")
candidates = [
    ("linear, basic", LinearRegression(), basic),
    ("linear, forecast and festival", LinearRegression(), richer),
    ("forest, forecast and festival", RandomForestRegressor(200, min_samples_leaf=5, random_state=0), richer),
]
results = []
for name, model, features in candidates:
    model.fit(train[features], train["target"])
    predictions = model.predict(valid[features])
    error = mae(predictions, valid)
    results.append((error, name, model, features, predictions))
    print(f"Validation MAE  {name}: {error:.1f}")

_, name, chosen, features, predictions = min(results, key=lambda result: result[0])
# Classroom cost assumptions. One missed high-pollution day costs five false alarms.
def alert_cost(cutoff):
    alerts = predictions > cutoff
    high = valid["target"] > 90
    return 5 * (high & ~alerts).sum() + (~high & alerts).sum()

threshold = min([90, 80, 70], key=alert_cost)
print(f"Selected on validation: {name}, threshold {threshold}")
# Freeze the model and decision rule before inspecting the test year.
pred = chosen.predict(test[features])
print(f"Test MAE  model: {mae(pred, test):.1f}   persistence: {mae(test['pm25'].values, test):.1f}")
bad = test["target"] > 90
alert = pred > threshold
print(f"Test recall: {(alert & bad).sum() / bad.sum():.2f}   false alarms: {int((alert & ~bad).sum())}")

Output

Validation MAE  persistence: 11.4
Validation MAE  linear, basic: 10.6
Validation MAE  linear, forecast and festival: 8.1
Validation MAE  forest, forecast and festival: 9.7
Selected on validation: linear, forecast and festival, threshold 80
Test MAE  model: 8.8   persistence: 11.7
Test recall: 0.99   false alarms: 24

Try it yourself

Choosing the alert threshold with principals

In this separate fictional validation exercise, five principals compare alert thresholds for a previous winter.

Good practice or bad practice?

Judge each step in improving the model.

  • Reporting only the metric that looks best

  • Trying 50 models on the test set and reporting the best

  • Changing one feature at a time and checking validation MAE

  • Checking errors separately for festival days

Common mistakes

  • Making several changes at once.
  • Tuning on the test set.
  • Choosing a threshold without asking the people who act on it.

Check yourself

Question 1

1. What is an ablation test?

Question 2

2. Lowering the alert threshold from 90 to 70 usually…

Question 3

3. Which data should guide choices between model versions?

Summary

  • Improve with targeted, one-at-a-time changes judged on validation data.
  • Evaluate once on untouched test data with several metrics, error analysis and comparison to the baseline.
  • Choose decision thresholds with users, balancing missed events and false alarms.

Finished reading?

Sources: Google ML Crash Course, monitoring pipelines · scikit-learn: model evaluation · Google: Rules of Machine Learning