Definition
Can you beat the baseline honestly, and explain how?
This project delivers a complete tabular prediction system: a framed problem with a cost-aware metric, a leak-free preprocessing pipeline, a baseline, several model families compared with appropriate cross-validation, a tuned final model, a threshold chosen from costs, one honest test evaluation, interpretability analysis and a written model card.
In simple words
Everything in this module, in one notebook. The example below predicts malignant tumours, where missing a malignant case is far worse than a false alarm. Replace the dataset with your own, such as loan default or customer churn, and follow the same steps.
Why it matters
Employers and reviewers judge ML skill by complete, honest projects rather than isolated algorithms. A documented, reproducible project with a model card is the standard portfolio piece.
Key terms
- Problem statement
- A precise description of the target, the decision supported, the users and the success metric.
- Feature engineering
- Creating informative inputs from raw data, such as ratios, counts and time-based features.
- Model selection
- Comparing candidate models with cross-validation and choosing one for the job.
- Error analysis
- Studying where and why the model fails, by segment, feature range and example.
- Model card
- Documentation of intended use, data, performance by group, limits and contacts.
- Serving and monitoring
- Exposing the model through an API and tracking inputs, outputs and accuracy over time.
Explained in detail
The project at a glance
This capstone builds a complete system that predicts whether a customer of a mobile-recharge company will stop using the service in the next month (churn), so the retention team can offer help to those at risk. It brings together every topic in the module. The target is defined precisely: no recharge for 30 days after the prediction date. Features are computed only from information available on that date, such as recharge frequency in the last 90 days, the trend in data usage, the number of complaints and the tenure of the customer. The split is by time, training on earlier months and testing on the latest month, to mirror deployment and avoid leakage.
Steps and deliverables
| Step | Deliverable |
|---|---|
| Frame and explore | Problem statement, data profile, chosen metric (recall at a fixed contact budget) |
| Prepare | A ColumnTransformer pipeline for imputation, encoding and scaling |
| Baseline | Rule "no recharge in 20 days" and a logistic regression |
| Model and tune | Random forest and gradient boosting tuned with time-series cross-validation |
| Evaluate and explain | Test results, PR curve, threshold choice, SHAP summary, errors by region |
| Deploy and document | Saved pipeline, FastAPI endpoint, model card, monitoring plan |
The business constraint shapes the evaluation. If the retention team can call 2,000 customers a week, the useful question is how many of the true churners appear among the 2,000 highest-risk customers, so the key metric is recall or precision at that cut-off, not overall accuracy. Error analysis might reveal that the model misses customers who switch to a competitor after a single bad network experience, suggesting a new feature from network-quality logs. A fairness check confirms that recall is similar across regions and age groups, so no group is systematically ignored by the retention offers.
- Retrain on a schedule and compare each new model with the current one on the latest data before switching.
- Monitor input drift, such as a new recharge plan changing usage patterns, and the actual churn of contacted and uncontacted customers.
- Measure business impact with an experiment: offer retention to a random half of high-risk customers and compare churn.
How it works
Milestones and rubric
- Problem framing: target, users, metric and error costs written down (10 per cent).
- Data and leakage audit: what is known at prediction time, splits that mimic use (15 per cent).
- Pipeline and baseline: preprocessing in a Pipeline, a trivial and a simple model (15 per cent).
- Model comparison and tuning with proper cross-validation (20 per cent).
- Threshold from costs, one test evaluation, error analysis (20 per cent).
- Interpretability and a model card: intended use, data, metrics, limits, risks (20 per cent).
In code
A compact end-to-end run: baseline, two tuned candidates, a cost-based threshold on validation folds, one test evaluation and the top drivers.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, recall_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_predict, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X, y = data.data, 1 - data.target # 1 = malignant
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)
COST_FP, COST_FN = 1, 20 # a false alarm against a missed malignancy
def cost(y_true, pred):
tn, fp, fn, tp = confusion_matrix(y_true, pred).ravel()
return fp * COST_FP + fn * COST_FN
print("Baseline cost (never flag):", cost(y_te, DummyClassifier().fit(X_tr, y_tr).predict(X_te)))
candidates = {
"logistic": GridSearchCV(make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)), {"logisticregression__C": [0.1, 1, 10]}, cv=cv, scoring="roc_auc"),
"boosting": GridSearchCV(HistGradientBoostingClassifier(random_state=0), {"max_depth": [2, 4], "learning_rate": [0.05, 0.1]}, cv=cv, scoring="roc_auc"),
}
for name, search in candidates.items():
search.fit(X_tr, y_tr)
print(f"{name:>9}: CV ROC-AUC {search.best_score_:.3f} {search.best_params_}")
best_name = max(candidates, key=lambda n: candidates[n].best_score_)
best = candidates[best_name].best_estimator_
oof = cross_val_predict(best, X_tr, y_tr, cv=cv, method="predict_proba")[:, 1]
threshold = min(np.linspace(0.05, 0.6, 56), key=lambda t: cost(y_tr, (oof >= t).astype(int)))
pred = (best.fit(X_tr, y_tr).predict_proba(X_te)[:, 1] >= threshold).astype(int)
print(f"Chosen: {best_name}, threshold {threshold:.2f}")
print(f"Test: recall {recall_score(y_te, pred):.3f}, cost {cost(y_te, pred)}, confusion {confusion_matrix(y_te, pred).ravel().tolist()}")
imp = permutation_importance(best, X_te, y_te, n_repeats=5, random_state=0).importances_mean
print("Top drivers:", [str(data.feature_names[j]) for j in np.argsort(imp)[::-1][:3]])Output
Baseline cost (never flag): 1060
logistic: CV ROC-AUC 0.995 {'logisticregression__C': 0.1}
boosting: CV ROC-AUC 0.988 {'learning_rate': 0.1, 'max_depth': 2}
Chosen: logistic, threshold 0.11
Test: recall 1.000, cost 16, confusion [74, 16, 0, 53]
Top drivers: ['worst radius', 'radius error', 'worst texture']Try it yourself
Write the model card
Fill in each section for your project in one or two sentences.
- Intended use and users: who uses the predictions, for what decision?
- Data: source, time period, size, known gaps and who is under-represented.
- Metrics: test results with uncertainty, and results per important group.
- Threshold and costs: what was chosen and why.
- Limits and risks: where the model should not be used, and what could go wrong.
Common mistakes
- Tuning the threshold on the test set.
- Comparing models on different splits.
- Skipping the model card, so nobody knows the model's limits.
Check yourself
Summary
- Frame, audit, pipeline, baseline, compare, tune, threshold, test once, explain, document.
- Costs drive the metric and the threshold.
- A model card makes the work usable and responsible.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: Mitchell et al., Model Cards for Model Reporting (2019)