I11 · Intermediate Capstone: An End-to-End AI Project

Building a Baseline Model

A baseline is the simplest reasonable method for a task, used as a reference point. Common baselines are predicting the average of the training data, predicting the most common class, a persistence forecast that repeats the latest value, or a simple rule from domain knowledge. A first machine-learning model, such as linear or logistic regression on a few features, is then compared against it on the same properly split data. Every later improvement is judged by how much it beats the baseline, not by its score in isolation.

20 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Before training anything clever, ask: how well would a lazy guess do? For air pollution, "tomorrow will be like today" is surprisingly hard to beat. If your fancy model cannot beat it, it is not useful yet.

A baseline is the simplest reasonable method for a task, used as a reference point. Common baselines are predicting the average of the training data, predicting the most common class, a persistence forecast that repeats the latest value, or a simple rule from domain knowledge. A first machine-learning model, such as linear or logistic regression on a few features, is then compared against it on the same properly split data. Every later improvement is judged by how much it beats the baseline, not by its score in isolation.

In simple words

Before claiming your new shoes make you run faster, time yourself in your old shoes. The baseline is the time in the old shoes; only the difference tells you whether the new shoes help.

Overview

  • A baseline is the simplest reasonable model or rule that your project must beat.
  • Without a baseline you cannot tell whether a complex model is actually helping.
  • Baselines are quick to build, easy to explain and reveal data problems early.
  • Always split data into training and test sets before building any model.
  • Use the same metric and test set for the baseline and later models.
  • A surprisingly strong baseline sometimes turns out to be the best solution.

Why it matters

Without a baseline, a score such as "mean error of 20" means nothing. Baselines catch bugs, show whether machine learning is worth the effort and set realistic expectations for stakeholders.

Key terms

Baseline
The simplest sensible method, used to judge whether a model adds value.
Persistence forecast
Predicting that the next value equals the latest one.
Time-based split
Training on earlier dates and testing on later ones, so the model never sees the future.
Mean absolute error (MAE)
The average size of prediction errors, in the target's units.
Skill score
How much a model reduces error relative to the baseline, such as 1 − MAE_model ÷ MAE_baseline.
Reproducibility
Fixed random seeds, saved data splits and code so results can be repeated.

Types

There are several types of baseline. Projects usually try two or three, from simplest upward.

1. Majority-class and mean baselines

Always predicting the most common label, or the average value.

  • How it works: scikit-learn's DummyClassifier and DummyRegressor compute these automatically.
  • Example: If 85% of emails are not spam, predicting "not spam" always gives 85% accuracy.
  • When to use it: Use it first to learn the minimum score any useful model must exceed. A model that cannot beat it has learned nothing useful.

2. Rule-based (heuristic) baselines

Simple hand-written rules based on common sense or domain knowledge.

  • How it works: Write if-else rules or simple formulas using one or two key features.
  • Example: Predicting next week's sales as the average of the last four weeks.
  • When to use it: Use it when experts already have a rule of thumb your model should beat.

3. Simple linear models

Linear or logistic regression on the main features.

  • How it works: Fit with default settings after basic scaling and encoding; inspect coefficients to understand the data.
  • Example: A logistic regression predicting pass or fail from attendance and homework scores with 78% accuracy.
  • When to use it: Use it as the standard ML baseline; it is fast, stable and interpretable.

4. Simple tree baselines

A shallow decision tree or a default random forest.

  • How it works: Train with default or limited depth to capture non-linear patterns without tuning.
  • Example: A depth-3 tree revealing that crop yield depends mostly on rainfall above a threshold.
  • When to use it: Use it to check whether non-linear models are likely to help before investing in tuning.

5. Pre-trained and zero-shot baselines

Using an existing model or LLM without any training on your data.

  • How it works: Apply a pre-trained classifier or prompt an LLM with clear instructions, then score it on your test set.
  • Example: Asking an LLM to label customer reviews as positive or negative and getting 88% accuracy.
  • When to use it: Use it for text, image and audio projects, where pre-trained models are often strong.

Explained in detail

Splitting time-series data correctly

With data that runs through time, a random split lets the model peek at the future: if 15 January is in the test set but 14 and 16 January are in training, the model effectively sees the answer. Instead, split by date: for example, train on the first two years and test on the most recent year, which mimics real use, where the model only knows the past. If you tune settings, hold out a validation period between training and test, or use rolling-origin evaluation that repeatedly trains on all data up to a date and tests on the following weeks. Keep the test period untouched until the end, and record the exact dates of each split.

Baselines for our project

BaselinePrediction for tomorrowWhat it tells us
Training meanThe average PM2.5 over the training yearsAny useful model must beat this easily
Seasonal meanThe average for this month in past yearsCaptures the winter peak
PersistenceToday's PM2.5Usually the hardest simple baseline for pollution
Linear regressionWeighted sum of today's PM2.5, the 3-day average, wind and humidityOur first real model

Evaluate all of them on the same test period with the same metrics. For regression, report mean absolute error, which is easy to explain ("on average we are 18 µg/m³ off"), and also root mean squared error, which punishes big misses. Because the decision is an alert, also convert predictions to "Poor or worse" yes or no and report recall and precision for those days. If linear regression barely beats persistence, that is a valuable finding: it tells you that most of the predictable signal is simply persistence, and the next page explores what extra information might help. Save the baseline results in a table that every later experiment adds a row to.

Baselines also protect you from fooling yourself. In our example, persistence already catches 96 per cent of Poor-or-worse days, because winter pollution stays high for weeks. That means the real value of a model lies in the difficult days, when the level changes sharply, and it tells us to judge improvements on those days, not only on the yearly average. Writing this insight down early keeps the project honest and focused on what users need.

How it works

  1. Split the data by date into training, validation and test periods.
  2. Compute simple baselines: mean, seasonal mean and persistence.
  3. Train a simple model such as linear regression on a few features.
  4. Evaluate all on the test period with MAE, RMSE and alert recall and precision.
  5. Record results in an experiment table with seeds, dates and settings.

In code

baseline.py
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def make_city_data(days=3 * 365, seed=3):
    rng = np.random.default_rng(seed)
    d = pd.date_range("2022-01-01", periods=days, freq="D")
    winter = np.cos((d.dayofyear - 15) / 365 * 2 * np.pi)            # +1 in mid-January, -1 in mid-July
    wind = np.clip(rng.normal(3 - 1.0 * winter, 1.2), 0.3, None)       # calmer air in winter
    humidity = np.clip(rng.normal(60 + 15 * winter, 10), 20, 100)
    festival = ((d.month == 11) & (d.day <= 3)).astype(int)
    pm = np.zeros(days); pm[0] = 120
    for t in range(1, days):
        level = 85 + 60 * winter[t] - 14 * (wind[t] - 3) + 0.3 * (humidity[t] - 60) + 150 * festival[t]
        pm[t] = 0.45 * pm[t - 1] + 0.55 * level + rng.normal(0, 10)
    return pd.DataFrame({"pm25": np.clip(pm, 10, None), "wind": wind, "humidity": humidity, "festival": festival}, index=d)

df = make_city_data()
df["target"] = df["pm25"].shift(-1)                                    # tomorrow's value
df["pm25_3day"] = df["pm25"].rolling(3).mean()
df["season"] = np.cos((df.index.dayofyear - 15) / 365 * 2 * np.pi)
df = df.dropna()
train, test = df[df.index < "2024-01-01"], df[df.index >= "2024-01-01"]   # split by date

def scores(name, pred):
    mae = np.mean(np.abs(pred - test["target"]))
    bad, alert = test["target"] > 90, pred > 90                        # "Poor or worse"
    recall = (alert & bad).sum() / bad.sum()
    false_alarms = int((alert & ~bad).sum())
    print(f"{name:<18} MAE={mae:5.1f}   alert recall={recall:.2f}   false alarms={false_alarms}")
    return mae

print("Poor-or-worse days in the 2024 test year:", int((test["target"] > 90).sum()))
scores("training mean", np.full(len(test), train["target"].mean()))
base = scores("persistence", test["pm25"].values)
features = ["pm25", "pm25_3day", "wind", "humidity", "season"]
lin = LinearRegression().fit(train[features], train["target"])
mae = scores("linear regression", lin.predict(test[features]))
print(f"Skill vs persistence: {1 - mae / base:.0%} lower error")

Output

Poor-or-worse days in the 2024 test year: 170
training mean      MAE= 51.2   alert recall=0.00   false alarms=0
persistence        MAE= 11.7   alert recall=0.96   false alarms=6
linear regression  MAE= 10.7   alert recall=0.96   false alarms=6
Skill vs persistence: 8% lower error

Try it yourself

Beating "tomorrow equals today"

The team runs its first comparison on the 2024 test year.

Correct split or leakage?

Judge each way of splitting the time-series data.

  • Shuffle all days and split 80/20 at random

  • Use today's measured wind and tomorrow's wind forecast

  • Use tomorrow's measured wind as a feature

  • Train on 2022–2023, test on 2024

Common mistakes

  • Reporting a model's score without any baseline.
  • Splitting time-series data at random.
  • Testing many ideas on the test set until one looks good.

Check yourself

Question 1

1. What is a persistence forecast?

Question 2

2. Why split time-series data by date?

Question 3

3. Persistence MAE is 20 and the model's MAE is 15. The skill is…

Summary

  • A baseline is the simplest sensible method; every model must beat it on the same test data.
  • For time series, split by date and keep the test period untouched.
  • Report MAE, RMSE and alert recall and precision in an experiment table.

Finished reading?

Sources: Google: Rules of Machine Learning · scikit-learn: model evaluation