A1 · Supervised Machine Learning in Depth

The End-to-End Supervised Workflow

The supervised learning workflow is the disciplined sequence used to build a predictive model: frame the problem and its success metric, establish a baseline, split the data so that the test set stays untouched, train candidate models, select and tune them on validation data, evaluate once on the test set, and iterate based on error analysis.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

How do you know a model is ready, and not just lucky on one split?

The supervised learning workflow is the disciplined sequence used to build a predictive model: frame the problem and its success metric, establish a baseline, split the data so that the test set stays untouched, train candidate models, select and tune them on validation data, evaluate once on the test set, and iterate based on error analysis.

In simple words

Treat the test set like a sealed exam paper. You may practise as much as you like on training data and rehearse on validation data, but you open the exam only once. A baseline, such as always predicting the most common class, tells you whether your model is actually learning anything.

Why it matters

Most failed ML projects fail in the workflow, not the algorithm: the wrong metric, a test set used for tuning, or no baseline to compare against. A clean workflow is what makes a reported score trustworthy.

Key terms

Supervised learning
Learning a mapping from inputs X to a known target y using labelled examples.
Target and features
The target is what we predict; features are the measured inputs used to predict it.
Train, validation and test sets
Data for fitting, for choosing settings, and for a final unbiased check.
Baseline
The simplest sensible predictor, such as the mean or the majority class, used as a reference.
Generalisation
Performing well on new data drawn from the same situation, not just on training data.
Model monitoring
Tracking a deployed model's inputs, outputs and accuracy to detect drift and failures.

Explained in detail

The stages of a supervised project

Every supervised project, whether it predicts loan default, crop yield or hospital readmission, moves through the same stages. First, frame the problem: what decision will the prediction support, what exactly is the target, and which metric reflects the cost of mistakes? Second, collect and understand the data with exploratory analysis. Third, split the data before doing anything that learns from it, so that the test set stays untouched. Fourth, prepare features: impute missing values, encode categories, scale numbers and engineer new features, fitting every step only on training data. Fifth, train a baseline and then candidate models, compare them with cross-validation on the training data, and tune hyperparameters. Sixth, evaluate the chosen model once on the test set and analyse its errors by group and condition. Seventh, deploy it behind an interface, document it in a model card, and monitor it for data drift and falling accuracy.

Choosing the right metric first

SituationSensible metricWhy
Predicting house pricesMean absolute error in rupeesEasy to explain; robust to a few extreme houses
Screening for a rare diseaseRecall at a fixed precision, or PR-AUCMissing a patient is far worse than an extra test
Spam filteringPrecision at high recallLosing a real email is costly
Ranking leads for a sales teamPrecision at top kOnly the top of the list is acted on

A common failure is optimising a metric that does not match the real goal, such as accuracy on data where 98 per cent of cases are negative. Another is iterating on the test set until it looks good, which turns it into a second validation set and makes the final score optimistic. Keep a written log of every experiment, fix random seeds and keep data versions, so results can be reproduced. Finally, remember that the model is only one part of the system: the data pipeline, the user interface, human review and monitoring often decide whether the project succeeds.

Consider a microfinance lender predicting which applicants will repay. The target is repayment within twelve months; features include income, existing loans and repayment history; the metric is recall of defaulters at an approval rate the business can afford. The data is split by application date, a logistic-regression baseline is trained, then gradient-boosted trees are tuned with cross-validation. The chosen model is tested once on the latest quarter, checked for differences between rural and urban applicants, and deployed with human review of borderline cases.

How it works

  1. Frame: what is predicted, for whom, and which metric reflects the real cost of errors?
  2. Split: train / validation / test (for example 60 / 20 / 20), stratified for classification.
  3. Baseline: a trivial model, such as the majority class or the mean.
  4. Train candidates and compare them on validation data or with cross-validation.
  5. Tune the best, then evaluate once on the test set and report that number.
  6. Analyse errors: which cases fail, and why? Then iterate on data and features.

In code

A complete workflow on the breast cancer dataset: stratified splits, a baseline, a candidate, validation-based selection and one final test score.

workflow.py
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, recall_score

X, y = load_breast_cancer(return_X_y=True)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=0)
print("Train / val / test sizes:", len(X_train), len(X_val), len(X_test))

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("Baseline validation accuracy:", round(accuracy_score(y_val, baseline.predict(X_val)), 3))

candidates = {C: make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=1000)) for C in [0.01, 1, 100]}
scores = {C: accuracy_score(y_val, m.fit(X_train, y_train).predict(X_val)) for C, m in candidates.items()}
for C, s in scores.items():
    print(f"C = {C:<6} validation accuracy {s:.3f}")
best_C = max(scores, key=scores.get)

final = make_pipeline(StandardScaler(), LogisticRegression(C=best_C, max_iter=1000)).fit(X_temp, y_temp)
pred = final.predict(X_test)
print("Chosen C:", best_C)
print("Test accuracy:", round(accuracy_score(y_test, pred), 3), "| test recall:", round(recall_score(y_test, pred), 3))

Output

Train / val / test sizes: 341 114 114
Baseline validation accuracy: 0.623
C = 0.01   validation accuracy 0.965
C = 1      validation accuracy 0.991
C = 100    validation accuracy 0.974
Chosen C: 1
Test accuracy: 0.982 | test recall: 1.0

Try it yourself

Order the workflow

Put the stages of a trustworthy supervised workflow in order.

  1. 1Split off a test set and lock it away
  2. 2Train and compare candidates on validation data
  3. 3Analyse errors and iterate
  4. 4Evaluate once on the test set
  5. 5Tune the best candidate
  6. 6Build a baseline
  7. 7Frame the problem and choose the metric

Common mistakes

  • Tuning on the test set, which turns it into a second validation set and inflates the score.
  • Skipping the baseline, so 95 per cent accuracy looks good even when 94 per cent is trivial.
  • Choosing a metric for convenience rather than for the real cost of errors.

Check yourself

Question 1

1. How often should the test set be used?

Question 2

2. Why stratify the split in classification?

Question 3

3. A baseline predicting the majority class scores 63 per cent. Your model scores 64 per cent. What does this suggest?

Summary

  • Frame, split, baseline, train, tune, test once, analyse.
  • Keep the test set sealed until the final evaluation.
  • Always compare against a baseline.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Common pitfalls