Definition
A house-price app scores well on last year's sales. On launch day one buyer leaves the area box empty, another's form sends 1200 as text, and a third lives in a locality the model has never seen. Which of these will crash the app, and which will quietly show a wrong price?
Testing an ML prediction pipeline means writing automated checks that send known inputs through every step, from input validation and preprocessing to the model and its output, and verify the behaviour promised for each kind of input: valid, missing, unseen and malformed. Tests check behaviour; a separate evaluation on held-out data measures accuracy.
In simple words
Before a lift opens to the public, inspectors don't only ride it to the right floor. They also press two buttons at once, overload it and cut the power, and check that it fails safely. Pipeline tests do the same for a prediction: the normal requests, and the awkward ones real users will certainly send.
Overview
- A prediction pipeline is a chain: check the input, preprocess it, run the model, format the output. A bug in any link reaches the user.
- Test the public entry point the app actually calls, not only the model inside it.
- Cover four kinds of input: valid, missing, unseen category and malformed or wrong type, plus the exact boundaries.
- Contract tests check exact promises; behavioural tests check properties such as 'a bigger home never costs less'.
- Regression tests keep every bug that once reached users, so it can never return unnoticed.
- Accuracy is a separate question measured on held-out data, and after launch monitoring watches for drift and training-serving skew.
Why it matters
Most of a real ML system is not the model. In 2015 D. Sculley and colleagues at Google described how the model code is only a small part of a production system; the rest is data handling, configuration and serving, where many failures happen. A model with an excellent test score can still crash on an empty field or multiply a string, and users meet those failures first.
Key terms
- Pipeline
- The ordered steps that turn a raw request into a prediction: validation, preprocessing, model and output formatting.
- Input contract
- The written promise of what the pipeline accepts, in which types and units, and what it does with anything else.
- Unit test
- A small automated check of one function's behaviour for one input.
- Fixture
- A fixed, repeatable input or setup that tests reuse, such as one known listing.
- Assertion
- A line that states the expected result and fails the test if the actual result differs.
- Edge case
- An input at or beyond the limits, such as zero area, a blank locality or an enormous number.
- Unseen category
- A category value that never appeared in the training data, such as a new locality.
- Behavioural test
- A test of a property the model should keep, such as invariance to spacing or a price that rises with area, without knowing the exact output.
- Regression test
- A saved test for a fixed bug, which fails if the bug ever comes back.
- Training-serving skew
- A difference between how data was processed during training and how it is processed for live requests.
- Data drift
- A change over time in the inputs a model receives, compared with its training data.
Types
A trustworthy pipeline is checked by several kinds of test, each answering a different question. Use all of them; none replaces another.
1. Input-contract tests
Tests that check the exact promise for each kind of input: valid, missing, unseen category, and malformed or wrong type.
- How it works: Call the public entry point with a small fixture and assert either the exact expected output or the specific exception that the contract names.
- Example: 100 sq ft in Market must return 5,00,000 rupees, while None, '1200', 0, -1, NaN, infinity and True must each raise ValueError.
- When to use it: First, for every pipeline, because these are the inputs real users send on day one.
2. Preprocessing tests
Tests of the cleaning, encoding and scaling steps between the raw request and the model.
- How it works: Feed raw values, compare the transformed result with the expected one, and check that training and serving use the same saved transformer object.
- Example: ' MARKET ' and 'market' must become the same encoded locality, and an unseen locality must become all zeros instead of an error.
- When to use it: Whenever a pipeline cleans text, fills gaps, encodes categories or scales numbers.
3. Behavioural tests
Tests of properties that a sensible model keeps without fixing its exact output: invariance and directional expectation.
- How it works: Change the input in a controlled way and check that the output stays the same, or moves in the expected direction.
- Example: Extra spaces in a locality must not change the price, and a 120 sq ft home must cost more than a 100 sq ft home in the same locality.
- When to use it: For trained models, whose exact numbers change after retraining but whose behaviour should not.
4. Regression tests
Saved tests for bugs that once happened, so that the same bug cannot return unnoticed.
- How it works: Reproduce the bug with the smallest input, write a test that fails, fix the code, and keep the test in the suite.
- Example: A form once sent the area as the text '1200', which Python repeated instead of multiplying; a test now checks that it raises ValueError.
- When to use it: Every time a user, a tester or a monitoring alert reports a bug.
5. Held-out evaluation
Measuring how accurate predictions are on data the model never saw during training.
- How it works: Compute an error metric such as mean absolute error on held-out sales, overall and for slices such as each locality, and compare it with the current model.
- Example: A new model may replace the old one only if its error on last month's sales is lower overall and no worse for any locality.
- When to use it: Before replacing a model, and regularly afterwards; tests alone cannot measure accuracy.
6. Monitoring in production
Continuous checks on live requests and predictions after the model is launched.
- How it works: Track the share of rejected inputs, the distribution of inputs and predictions, and the error once real sale prices arrive, with alerts for large changes.
- Example: If requests from new localities jump from 2 to 20 per cent, the fallback rate is being used far more than planned and the model may need retraining.
- When to use it: From launch onwards, because data drift can make a correct pipeline give wrong answers.
Explained in detail
Where a prediction can go wrong
Imagine a buyer typing a floor area and a locality into a house-price form. Before a price appears, the request passes through several steps: the input is checked, the locality is cleaned and encoded the way it was during training, the model computes a number, and the number is formatted in rupees. A test that calls only the model's final calculation misses everything before it. Start from the function the form actually calls, and for each kind of input write down what the buyer should see before writing any code.
| Step | What can go wrong | Test that catches it |
|---|---|---|
| Input check | A blank, text or negative area is accepted | Contract tests with bad values |
| Cleaning and encoding | ' Market ' treated as a new locality, or a new locality crashes the encoder | Invariance test and unseen-category test |
| Model | A retrained model's price falls as area rises | Directional test |
| Output | Price shown in paise instead of rupees | Exact value test on a known listing |
| Live use | Training and serving clean data differently, or inputs drift | Skew checks and monitoring |
Write the contract before the tests
In this lesson the reference estimator multiplies the area by a fixed rate per square foot for each locality. Its contract says: the area must be a positive, finite number of square feet; a boolean or text value is rejected even when it looks numeric; a locality it has never seen uses a published fallback rate of 3,500 rupees per square foot; and a blank locality is rejected, because the buyer supplied no name at all. A missing area is also rejected, since filling it with zero or the average would show a price that looks real but means nothing. The check uses type() rather than isinstance(), because in Python True counts as the integer 1 and would otherwise be priced as a one-square-foot home. Another product could make different choices, but its tests must state those choices explicitly.
Expected answers you can check by hand
Choose fixtures small enough to work out independently: 100 square feet at 5,000 rupees per square foot is 5,00,000 rupees, or five lakh. Never compute the expected value by calling the function you are testing, or a bug will appear on both sides of the comparison and the test will still pass. Testers call the source of expected answers the oracle, and a hand calculation is the most trustworthy one. Decimal prices need care, because computers store most decimals approximately: in Python, 0.1 + 0.2 gives 0.30000000000000004. Compare such results with math.isclose or assertAlmostEqual, and compare whole-rupee results exactly.
Testing a trained model: properties instead of exact numbers
When the fixed rates are replaced by a model trained on sales, a retrained model may give a different but equally reasonable price for the same house, so exact-value tests break for no good reason. Test properties that any sensible model should keep. Marco Tulio Ribeiro and colleagues called this behavioural testing in their CheckList paper, which won the best paper award at ACL 2020, and named three kinds: minimum functionality tests, simple cases that must be right; invariance tests, changes that must not change the output, such as extra spaces in a locality; and directional expectation tests, changes that must move the output one way, such as a larger area raising the price. The paper was about language models, but the ideas apply to any model.
Preprocessing deserves its own tests. In scikit-learn, put the encoder and the model in one Pipeline and save that whole object, so that serving uses exactly the cleaning learned during training; Google's Rules of Machine Learning call a mismatch between the two training-serving skew. An encoder created with OneHotEncoder(handle_unknown='ignore') turns an unseen locality into all zeros instead of crashing, and a test should confirm both that it does not crash and that the result is still a finite price in rupees.
Every bug becomes a regression test
Suppose a form once sent the area as the text '1200'. In Python, multiplying text by a number repeats the text, so '1200' * 4000 is a string 16,000 characters long, not a price, and the app showed nonsense. The fix is the strict type check, and the lasting lesson is the test. Reproduce the bug with the smallest input, write a test that fails, fix the code, watch the test pass and keep it forever. Running a suite of such tests after every change costs seconds, and a failing test names exactly which promise was broken.
Beyond the test suite: evaluation and monitoring
Passing tests show that the pipeline behaves as promised; they do not show that its prices are accurate. Measure accuracy separately on held-out sales with a metric such as mean absolute error, overall and for each locality, because a model can be accurate on average and poor for one area. In 2017 Eric Breck and colleagues at Google published the ML Test Score, a rubric of 28 tests covering data, model development, infrastructure and monitoring, as a checklist for whether a system is ready for production. After launch, watch the share of rejected requests, the mix of areas and localities and the predictions themselves: when live inputs drift away from the training data, a model can pass every test and still become wrong. Teams run the whole suite automatically on every change, and property-based testing tools such as Hypothesis generate hundreds of random inputs in search of one that breaks a property.
Run the code below and all 8 tests pass: 4 contract tests, 3 behavioural tests and 1 regression test. Its last line prices a 1,200 square foot home near the station at 48.0 lakh rupees, which you can confirm by hand: 1,200 × 4,000 = 48,00,000.
How it works
- Write the input contract: accepted types, units and ranges, the rule for unseen categories, and the error for anything else.
- Pick small fixtures whose expected outputs you can calculate by hand.
- Write contract tests for valid, missing, unseen and malformed inputs, using subTest for groups of bad values.
- Add behavioural tests: invariance to irrelevant changes and the expected direction of change.
- Turn every reported bug into a regression test before fixing it.
- Run the whole suite on every change, measure accuracy separately on held-out data, and monitor live inputs after launch.
The maths
- Rupees per square foot for locality ℓ, with 3,500 as the fallback
- The actual sale price of house i
- The predicted price of house i
Every expected value in the tests can be checked by hand. Market: 100 × 5,000 = 5,00,000 rupees. An unseen locality such as Lake uses the fallback: 100 × 3,500 = 3,50,000. The directional test compares 120 × 4,000 = 4,80,000 with 100 × 4,000 = 4,00,000 at the station, so the larger home costs more. A decimal area gives 100.1 × 4,000 = 4,00,400, which the test compares with math.isclose because the computer stores 100.1 approximately.
Accuracy is a different calculation. If three held-out sales are mispriced by 20,000, 50,000 and 10,000 rupees, the mean absolute error is (20,000 + 50,000 + 10,000) ÷ 3 ≈ 26,667 rupees. No unit test can produce this number, because it needs real sale prices.
In code
Three test classes separate the kinds of promise: exact contract behaviour, properties that should survive retraining, and a saved bug. Each bad value runs as its own subTest, so a failure names the exact input that slipped through.
import math
import unittest
RATES = {"market": 5000, "station": 4000} # rupees per square foot
FALLBACK = 3500 # published rate for any other locality
def predict_price(area, locality):
"""The function the listing form calls. Returns rupees or raises ValueError."""
if type(area) not in (int, float) or not math.isfinite(area) or area <= 0:
raise ValueError("Enter a positive numeric area in square feet")
if not isinstance(locality, str) or not locality.strip():
raise ValueError("Enter a locality name")
return area * RATES.get(locality.strip().lower(), FALLBACK)
class Contract(unittest.TestCase):
"""Minimum functionality: what each kind of input must produce."""
def test_known_listing(self):
self.assertEqual(predict_price(100, "Market"), 500_000)
def test_unseen_locality_uses_fallback(self):
self.assertEqual(predict_price(100, "Lake"), 350_000)
def test_bad_area_is_rejected(self):
for area in (None, "1200", 0, -1, float("nan"), float("inf"), True):
with self.subTest(area=area), self.assertRaises(ValueError):
predict_price(area, "Market")
def test_missing_locality_is_rejected(self):
for locality in ("", " ", None, 42):
with self.subTest(locality=locality), self.assertRaises(ValueError):
predict_price(100, locality)
class Behaviour(unittest.TestCase):
"""Properties any sensible price model should keep, whatever its exact numbers."""
def test_spacing_and_case_do_not_matter(self):
self.assertEqual(predict_price(100, " MARKET "), predict_price(100, "market"))
def test_bigger_home_costs_more(self):
for locality in ("Market", "Station", "Lake"):
with self.subTest(locality=locality):
self.assertGreater(predict_price(120, locality), predict_price(100, locality))
def test_decimal_area(self):
self.assertTrue(math.isclose(predict_price(100.1, "Station"), 400_400))
class Regression(unittest.TestCase):
"""Bugs that once reached users stay tested forever."""
def test_area_typed_as_text(self):
# Bug 12: the form sent "1200" as text, and "1200" * 4000 repeats the text 4000 times.
with self.assertRaises(ValueError):
predict_price("1200", "Station")
for case in (Contract, Behaviour, Regression):
result = unittest.TestResult()
unittest.defaultTestLoader.loadTestsFromTestCase(case).run(result)
print(f"{case.__name__:<10} tests run: {result.testsRun}, failed: {len(result.failures)}, crashed: {len(result.errors)}")
for test, trace in result.failures + result.errors:
print(" ", test.id().split(".")[-1], "->", trace.strip().splitlines()[-1])
print(f"1,200 sq ft near the station: {predict_price(1200, 'Station') / 100_000:.1f} lakh rupees")Output
Contract tests run: 4, failed: 0, crashed: 0 Behaviour tests run: 3, failed: 0, crashed: 0 Regression tests run: 1, failed: 0, crashed: 0 1,200 sq ft near the station: 48.0 lakh rupees
Try it yourself
Which kind of test is this?
Sort each check into the kind of test it belongs to.
' MARKET ' and 'market' must give the same price
A 120 sq ft home must cost more than a 100 sq ft home
The text '1200' that once broke the app must be rejected
100 sq ft in Market must cost exactly 5 lakh
Error on last month's 500 sales must stay below 40,000 rupees
A blank locality must raise ValueError
From bug report to safe fix
Put the steps for handling a reported bug in order.
- 1Run the whole suite to check nothing else broke
- 2Reproduce the bug with the smallest input
- 3Write a test that fails because of the bug
- 4Fix the code
- 5Keep the test in the suite permanently
- 6Run the new test and see it pass
Can your tests catch a change?
Edit the code in the editor and run after each change.
- Change FALLBACK to 3000. Predict which test fails and the two amounts its message will show. Run and check.
- Restore 3500. Replace the first check in predict_price with: if area is None or area == 0:. Predict how many failures appear and which inputs they name.
- Restore the original line. Add a contract test for 50 sq ft at Station, working out the expected price before you run it.
Think about it
- Which bug would users have met if the regression test did not exist? Which test would have caught it anyway?
Common mistakes
- Testing only the model's predict step and skipping input checks and preprocessing.
- Computing the expected value with the function under test, so a bug appears on both sides and the test still passes.
- Catching every exception in a test meant for one input error, so unrelated crashes pass too.
- Comparing decimal prices with exact equality instead of a tolerance.
- Filling a missing area with zero or the average so that a price always appears.
- Treating passing tests as proof of accuracy; accuracy needs held-out data, and drift needs monitoring after launch.
Check yourself
Summary
- Test the public entry point the app calls, through validation, preprocessing, model and output.
- Write the input contract first, and cover valid, missing, unseen and malformed inputs with hand-checked expectations.
- Use behavioural tests, invariance and direction, for trained models whose exact numbers change.
- Turn every bug into a regression test and run the whole suite on every change.
- Measure accuracy separately on held-out data, and monitor drift and skew after launch.
Finished reading?
Sources: Python documentation · unittest · Breck et al. · The ML Test Score: a rubric for ML production readiness (2017) · Ribeiro et al. · Beyond Accuracy: behavioral testing of NLP models with CheckList (ACL 2020) · scikit-learn · OneHotEncoder and handle_unknown · Google · Rules of Machine Learning · Hypothesis · property-based testing for Python