Definition
How do you know a model is ready, and not just lucky on one split?
The supervised learning workflow is the disciplined sequence used to build a predictive model: frame the problem and its success metric, establish a baseline, split the data so that the test set stays untouched, train candidate models, select and tune them on validation data, evaluate once on the test set, and iterate based on error analysis.
In simple words
Treat the test set like a sealed exam paper. You may practise as much as you like on training data and rehearse on validation data, but you open the exam only once. A baseline, such as always predicting the most common class, tells you whether your model is actually learning anything.
Why it matters
Most failed ML projects fail in the workflow, not the algorithm: the wrong metric, a test set used for tuning, or no baseline to compare against. A clean workflow is what makes a reported score trustworthy.
Key terms
- Supervised learning
- Learning a mapping from inputs X to a known target y using labelled examples.
- Target and features
- The target is what we predict; features are the measured inputs used to predict it.
- Train, validation and test sets
- Data for fitting, for choosing settings, and for a final unbiased check.
- Baseline
- The simplest sensible predictor, such as the mean or the majority class, used as a reference.
- Generalisation
- Performing well on new data drawn from the same situation, not just on training data.
- Model monitoring
- Tracking a deployed model's inputs, outputs and accuracy to detect drift and failures.
Explained in detail
The stages of a supervised project
Every supervised project, whether it predicts loan default, crop yield or hospital readmission, moves through the same stages. First, frame the problem: what decision will the prediction support, what exactly is the target, and which metric reflects the cost of mistakes? Second, collect and understand the data with exploratory analysis. Third, split the data before doing anything that learns from it, so that the test set stays untouched. Fourth, prepare features: impute missing values, encode categories, scale numbers and engineer new features, fitting every step only on training data. Fifth, train a baseline and then candidate models, compare them with cross-validation on the training data, and tune hyperparameters. Sixth, evaluate the chosen model once on the test set and analyse its errors by group and condition. Seventh, deploy it behind an interface, document it in a model card, and monitor it for data drift and falling accuracy.
Choosing the right metric first
| Situation | Sensible metric | Why |
|---|---|---|
| Predicting house prices | Mean absolute error in rupees | Easy to explain; robust to a few extreme houses |
| Screening for a rare disease | Recall at a fixed precision, or PR-AUC | Missing a patient is far worse than an extra test |
| Spam filtering | Precision at high recall | Losing a real email is costly |
| Ranking leads for a sales team | Precision at top k | Only the top of the list is acted on |
A common failure is optimising a metric that does not match the real goal, such as accuracy on data where 98 per cent of cases are negative. Another is iterating on the test set until it looks good, which turns it into a second validation set and makes the final score optimistic. Keep a written log of every experiment, fix random seeds and keep data versions, so results can be reproduced. Finally, remember that the model is only one part of the system: the data pipeline, the user interface, human review and monitoring often decide whether the project succeeds.
Consider a microfinance lender predicting which applicants will repay. The target is repayment within twelve months; features include income, existing loans and repayment history; the metric is recall of defaulters at an approval rate the business can afford. The data is split by application date, a logistic-regression baseline is trained, then gradient-boosted trees are tuned with cross-validation. The chosen model is tested once on the latest quarter, checked for differences between rural and urban applicants, and deployed with human review of borderline cases.
How it works
- Frame: what is predicted, for whom, and which metric reflects the real cost of errors?
- Split: train / validation / test (for example 60 / 20 / 20), stratified for classification.
- Baseline: a trivial model, such as the majority class or the mean.
- Train candidates and compare them on validation data or with cross-validation.
- Tune the best, then evaluate once on the test set and report that number.
- Analyse errors: which cases fail, and why? Then iterate on data and features.
In code
A complete workflow on the breast cancer dataset: stratified splits, a baseline, a candidate, validation-based selection and one final test score.
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, recall_score
X, y = load_breast_cancer(return_X_y=True)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=0)
print("Train / val / test sizes:", len(X_train), len(X_val), len(X_test))
baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("Baseline validation accuracy:", round(accuracy_score(y_val, baseline.predict(X_val)), 3))
candidates = {C: make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=1000)) for C in [0.01, 1, 100]}
scores = {C: accuracy_score(y_val, m.fit(X_train, y_train).predict(X_val)) for C, m in candidates.items()}
for C, s in scores.items():
print(f"C = {C:<6} validation accuracy {s:.3f}")
best_C = max(scores, key=scores.get)
final = make_pipeline(StandardScaler(), LogisticRegression(C=best_C, max_iter=1000)).fit(X_temp, y_temp)
pred = final.predict(X_test)
print("Chosen C:", best_C)
print("Test accuracy:", round(accuracy_score(y_test, pred), 3), "| test recall:", round(recall_score(y_test, pred), 3))Output
Train / val / test sizes: 341 114 114 Baseline validation accuracy: 0.623 C = 0.01 validation accuracy 0.965 C = 1 validation accuracy 0.991 C = 100 validation accuracy 0.974 Chosen C: 1 Test accuracy: 0.982 | test recall: 1.0
Try it yourself
Order the workflow
Put the stages of a trustworthy supervised workflow in order.
- 1Split off a test set and lock it away
- 2Train and compare candidates on validation data
- 3Analyse errors and iterate
- 4Evaluate once on the test set
- 5Tune the best candidate
- 6Build a baseline
- 7Frame the problem and choose the metric
Common mistakes
- Tuning on the test set, which turns it into a second validation set and inflates the score.
- Skipping the baseline, so 95 per cent accuracy looks good even when 94 per cent is trivial.
- Choosing a metric for convenience rather than for the real cost of errors.
Check yourself
Summary
- Frame, split, baseline, train, tune, test once, analyse.
- Keep the test set sealed until the final evaluation.
- Always compare against a baseline.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Common pitfalls