Evaluating Models

Models that looked great

In 2018 a team at Mount Sinai trained a network to spot pneumonia on chest X-rays. On held-out images it scored beautifully. Then they asked it a different question: which hospital did this X-ray come from? It got that right more than 99% of the time.

That second result was the warning. Hospitals differ in how sick their patients are, in scanner models, in the little metal tokens technicians place on the film. A network rewarded for predicting pneumonia will happily learn “this looks like the busy inpatient ward” if that correlates with the label. When the pneumonia model was tested on images from a hospital it had never seen, its performance fell (Zech et al., PLOS Medicine). The model had not learned the disease. It had learned the building.

This is the most important idea in this lesson: a model is only as good as the test you put it through, and most tests are easier than the real world. Evaluation is the discipline of building tests that are hard enough to be honest.

What is model evaluation?

Evaluation estimates how a model will perform on data it has not seen, under the conditions where it will actually be used. It has two halves: a protocol (which data you test on, and what you are allowed to look at) and a metric (which number you compute, and whether it measures what you care about).

How does it go wrong?

Almost every failure falls into a few patterns. The test data is too similar to the training data (the pneumonia model). The metric rewards the wrong behaviour (accuracy on rare events, as you will see shortly). Information from the test set leaks into training. Or the world changes after deployment. The Epic Sepsis Model, used at hundreds of US hospitals, was checked independently on 38,455 hospital stays at Michigan Medicine: it had an AUC of 0.63 and did not alert on 67% of patients who developed sepsis (Wong et al., JAMA Internal Medicine). During the pandemic, a review of 62 COVID-19 imaging studies found that none of the models were of potential clinical use, mostly because of flawed evaluation (Roberts et al.).

Why does it matter?

Every decision downstream, whether to ship, which model to pick, how much to trust a prediction, rests on the evaluation number. If that number is inflated, everything built on it is too. The same skills apply whether you are testing a spam filter, a medical model or a frontier language model; only the details change.

Key takeaways

• A score only means something relative to the test that produced it; easy tests produce flattering scores.
• Models exploit any shortcut that predicts the label, including ones that vanish in deployment.
• Evaluation = a protocol (what data, what you may look at) + a metric (what number, measuring what).

Train, validation, test

If you study using the exam paper, your exam mark tells nobody how much you know. Machine learning has the same problem, and the fix is to lock some data away.

What are the three splits?

The training set is what the model learns from. The validation set (sometimes called the development set) is what you use to make choices: which architecture, which learning rate, when to stop training. The test set is kept sealed until the very end and used once, to report how the final model performs.

Why do we need a third set?

Training error is useless as an estimate: a big enough model can memorise its training data and score perfectly. So we hold out validation data. But every time you look at the validation score and change something, you are fitting to it a little. Try enough variants and one of them will do well on those particular examples by luck. The demo below makes that luck visible.

With one model, both numbers hover around 50%. With a few hundred, the best validation score climbs towards 60% while the test score of the same model stays near 50%. Nothing was trained on the validation data, yet selecting on it was enough to inflate it. This is why the number you report must come from data that played no part in any decision.

The effect is real at scale too. When researchers carefully rebuilt a new ImageNet test set following the original recipe, accuracy of a wide range of published models dropped by 11 to 14 percentage points (Recht et al., “Do ImageNet classifiers generalize to ImageNet?”). Interestingly, they traced most of the drop to the new images being slightly harder rather than to a decade of adaptive overfitting, and models kept their ranking. A reminder that “same distribution” is harder to achieve than it sounds.

Splitting well

• Split by the unit you will generalise over. If one patient has ten scans, all ten go in the same split, or the model can recognise the patient instead of the disease.
• Respect time. For forecasting, train on the past and test on the future. Shuffling lets the model peek ahead.
• Match deployment. If the model will run at new hospitals, the test set should contain a hospital it never trained on.
• Stratify rare classes so every split contains enough positives to measure anything.

Key takeaways

• Train to fit, validate to choose, test once to report.
• Choosing the best of many models on the same data inflates its score even though nothing was trained on it.
• Split by patient, user, site or time, whatever the model must generalise across in deployment.

Why accuracy lies

A fraud detector that never flags anything is 99.8% accurate. Accuracy counts every example equally, so when one class is rare it mostly measures how good the model is at the common one.

What is class imbalance?

A problem is imbalanced when one outcome is much rarer than the other. It is the normal case for the problems that matter: fraud, disease, machine failures, security intrusions. In the widely used European card-fraud dataset, only 492 of 284,807 transactions are fraudulent, about 0.17% (Dal Pozzolo et al.).

How does the arithmetic work?

Take 100,000 transactions at that fraud rate: 172 frauds, 99,828 legitimate. Suppose a model catches 95% of fraud and correctly clears 99% of legitimate payments. That sounds excellent. It catches 163 frauds and misses 9. But 1% of 99,828 is 998 false alarms. Its accuracy is 98.99%, lower than the 99.83% you get by predicting “legitimate” every time. And of the 1,161 transactions it flags, only 163 (14%) are fraud.

Switch to the balanced scenario with the same model and precision jumps to about 99%. The model has not changed at all; only the base rate has. That is why a number like “95% accurate” is meaningless until you know how common the thing being detected is.

Why does it matter?

In medicine this is the reason screening tests for rare conditions produce many false positives, and why doctors confirm positive screens with a second test. In machine learning it means imbalanced problems need metrics that look at the rare class directly: precision, recall and the curves built from them. Those are the subject of the next two sections.

Key takeaways

• Always compare accuracy with the trivial baseline of predicting the majority class.
• When positives are rare, even a very specific model produces mostly false alarms.
• The same model has different precision in populations with different base rates.

The confusion matrix

Every binary prediction lands in one of four boxes. Almost every classification metric is just a different ratio of those four counts.

What is it?

A confusion matrix tabulates predictions against reality. True positives are positives the model caught. False negatives are positives it missed. False positives are false alarms. True negatives are negatives it correctly cleared. Which of the two error types is worse depends entirely on the application.

How do the metrics work?

Four ratios cover most practical needs:

Precision asks “when it says yes, is it right?” Recall asks “of everything it should find, how much does it find?”

F1 is the harmonic mean of precision and recall, which punishes a model that is great at one and terrible at the other. It ignores true negatives, which is useful when negatives are plentiful and uninteresting.

Where does the threshold come in?

Most classifiers do not output yes or no. They output a score, often a probability, and a threshold turns it into a decision. Moving the threshold trades one error for the other. There is no single “accuracy of the model”: there is a family of operating points, and you choose one.

Push the threshold towards 0 and recall reaches 100% while precision falls to the share of positives (24 of 60, 40%). Push it towards 1 and precision rises while recall collapses. Somewhere in the middle F1 peaks, but that middle is only “best” if the two errors cost the same. In cancer screening you accept many false alarms to avoid misses. In a spam filter a lost legitimate email costs more than a missed spam, so you lean the other way.

Key takeaways

• TP, FP, FN and TN are the raw material; precision, recall, specificity and F1 are ratios of them.
• A threshold turns scores into decisions, and moving it trades false alarms for misses.
• Pick the operating point from the real costs of each error, not by default at 0.5.

ROC and precision-recall curves

Instead of judging a model at one threshold, sweep through all of them and draw the whole trade-off. That is what ROC and precision-recall curves do.

What is an ROC curve?

The receiver operating characteristic curve plots true positive rate (recall) against false positive rate as the threshold moves from strict to lenient. A useless model follows the diagonal. A perfect one hugs the top-left corner. The area under the curve (AUC) has a lovely interpretation: it is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative (Fawcett, “An introduction to ROC analysis”). An AUC of 0.63, like the sepsis model above, means the model ranks a random sepsis patient above a random non-sepsis patient only 63% of the time.

What is a precision-recall curve?

It plots precision against recall over the same thresholds. Its baseline is not a diagonal but a flat line at the share of positives: a random model flagging items at random has precision equal to the base rate. Its summary number is average precision (AP), the area under the curve computed as a weighted mean of precision at each recall level.

At a separation of 1.5, AUC sits around 0.85 whatever the prevalence. The ROC curve barely moves when you go from 50% to 1% positives, because both of its axes are rates within a single class. The PR curve collapses: at 1%, most of what the model flags is negative, and AP falls to a fraction of its balanced value. For rare-event problems the PR view matches what users experience, which is why Saito and Rehmsmeier recommend it for imbalanced data (Saito & Rehmsmeier).

Key takeaways

• ROC AUC is the chance a random positive outranks a random negative; 0.5 is a coin flip.
• ROC ignores class balance, which makes it stable but can make rare-event models look better than they feel.
• Precision-recall curves expose false alarms among flagged items; the baseline is the positive rate.

Cross-validation and leakage

One validation split is one noisy measurement. Cross-validation takes several and averages them. But it only works if nothing from the held-out fold leaks into training.

What is k-fold cross-validation?

Split the data into k equal folds. Train on k − 1 of them and validate on the remaining one; repeat so every fold is held out once; average the k scores. Every example is used for validation exactly once, and the spread of fold scores tells you how much the estimate would wobble with a different split. Ten folds, stratified by class, is a well-tested default (Kohavi 1995).

Notice how much the fold scores differ: the fold that happens to hold out an outlier looks far worse. A single train/validation split would have reported one of those numbers, and you would not know which. Cross-validation gives you the mean and, just as important, the spread.

Leakage: when the test answers leak into training

What is leakage?

Data leakage is any path by which information that would not be available at prediction time reaches the model during training or selection. Kapoor and Narayanan surveyed the literature and found leakage documented in 294 papers across 17 scientific fields, sometimes producing wildly over-optimistic conclusions. When they corrected the errors in a set of civil-war prediction studies, complex machine-learning models no longer beat decades-old logistic regression (Kapoor & Narayanan, Patterns).

Common forms:

• Preprocessing (scaling, imputation, feature selection) fitted on all data before splitting.
• Duplicates or near-duplicates across splits: the same patient, user, product photo or paragraph of text.
• Features that encode the answer: a “treatment given” field when predicting diagnosis, a timestamp that follows the event.
• Training on the future when predicting the past, via shuffled time series.

The leaky pipeline reports accuracy far above chance on data that contains no signal at all, because its feature selection already saw every validation label. Moving the same step inside each fold brings the estimate back to about 50%, the truth (Hastie, Tibshirani & Friedman, section 7.10.2). The rule generalises: every step that learns from data, including preprocessing, belongs inside the cross-validation loop. Tools like scikit-learn’s Pipeline exist to make that automatic.

Key takeaways

• k-fold cross-validation uses every example for validation once and reports a mean and a spread.
• Any step fitted on data, even feature selection or scaling, must be fitted inside each training fold.
• Leakage is common in published science and can make noise look like a breakthrough.

Errors in numbers and probabilities

Not every model says yes or no. Some predict a price or a temperature, and some output a probability that people will act on. Both need their own kind of evaluation.

What are the regression metrics?

yᵢ is the true value, ŷᵢ the prediction, ȳ the mean of the true values.

Mean absolute error is the typical size of a miss, in the units you care about. Root mean squared error squares before averaging, so large misses dominate. R² compares your model with the laziest possible one, always predicting the mean: 1 is perfect, 0 is no better than the mean, and it can go negative when the model is worse than that.

With no big miss, MAE is about £9.6k and RMSE £10.5k: similar, because the errors are similar in size. Undershoot one house by £200k and MAE rises to about £34.6k, while RMSE jumps to about £71.5k. Neither is “right”. If one catastrophic miss is much worse than several small ones (a bridge load estimate), RMSE reflects that. If every pound of error costs the same (a pricing estimate over thousands of homes), MAE is the honest summary.

What is calibration?

A model is calibrated if, among all the cases where it says “70%”, the event happens about 70% of the time. Ranking metrics like AUC ignore this completely: multiply every score by 0.5 and AUC is unchanged, but the probabilities are now wrong. Guo and colleagues found that modern deep networks became more accurate but also more overconfident than older, smaller ones, and that a one-parameter fix called temperature scaling repairs much of it (Guo et al., “On calibration of modern neural networks”).

Why does it matter?

Whenever someone acts on the probability itself (a doctor weighing a 12% risk, a bank pricing a loan, an agent deciding whether to ask a human) a miscalibrated score leads to systematically bad decisions even if the ranking is excellent.

Key takeaways

• MAE is the typical miss; RMSE weights big misses heavily; R² compares against always predicting the mean.
• Calibration asks whether a “70%” really happens 70% of the time; AUC cannot see it.
• Choose metrics that match how predictions will be used, not whichever is conventional.

Evaluating language models

Large language models break most of the assumptions above. There is no single label, the outputs are open-ended text, and the training data is most of the public internet, possibly including your test set.

What changes for LLMs?

Classic benchmarks still exist: multiple-choice knowledge tests, maths word problems with a single numeric answer, coding problems checked by unit tests. Those give crisp, automatic scores. But much of what people want from an assistant (a helpful explanation, a good summary, a safe refusal) has no single right answer, so evaluation relies on human preferences or on another model acting as a judge.

Problem 1: contamination

If benchmark questions (or their answers) appear in the training data, the benchmark measures memory, not skill. It is the LLM version of leakage. To test this, Scale AI researchers wrote GSM1k, a fresh set of grade-school maths problems matched in style and difficulty to the popular GSM8K benchmark. Several model families scored up to 8 points worse on the new problems, and models that were more likely to reproduce GSM8K examples verbatim showed bigger gaps (Zhang et al., 2024). The best frontier models showed little or no gap, so contamination is a spectrum, not a scandal about everyone.

Problem 2: judging open-ended answers

Using a strong LLM to grade answers is cheap and scales. Zheng and colleagues found that GPT-4 as a judge agreed with human preferences over 80% of the time, about as often as humans agree with each other. They also documented its biases: it tends to favour the answer shown first (position bias), longer answers (verbosity bias) and answers written by itself (self-enhancement bias) (Zheng et al., “Judging LLM-as-a-judge”). Good practice swaps answer order, controls for length and checks the judge against human labels.

Problem 3: leaderboards as targets

Crowd-sourced arenas, where people vote between two anonymous chatbots, avoid fixed test sets. But any public number becomes a target. A 2025 analysis argued that some providers privately tested many model variants and published only the best, and that access to arena data was unequal, which can distort rankings (Singh et al., “The leaderboard illusion”). It is the winner’s curse from the splits section, again, at industrial scale.

Key takeaways

• Contamination is leakage at web scale; fresh or private test sets reveal it.
• LLM judges are useful but biased towards position, length and themselves.
• Any public leaderboard becomes a target; build a small, private eval for your own use case.

Check your understanding

Seven scenarios from real projects. Each asks what you would do, not what a term means.

A bank’s fraud model is 99.1% accurate on a test set where 0.2% of transactions are fraud. The product manager wants to ship it. What should you check first?

References

The papers and texts behind the numbers in this lesson. Each is worth reading in full.

References

1. [1]

   Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study(opens in a new tab)

   Zech, Badgeley, Liu, Costa, Titano & Oermann, 2018

   PLOS Medicine. CNNs identified the hospital a radiograph came from with over 99% accuracy and lost performance on external sites.

2. [2]

   External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients(opens in a new tab)

   Wong, Otles, Donnelly et al., 2021

   JAMA Internal Medicine. The Epic Sepsis Model reached an AUC of 0.63 on 38,455 hospitalisations and missed 67% of sepsis cases.

3. [3]

   Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans(opens in a new tab)

   Roberts, Driggs, Thorpe et al., 2021

   Nature Machine Intelligence. Of 62 studies that passed quality screening, none produced a model of potential clinical use.

4. [4]

   Do ImageNet classifiers generalize to ImageNet?(opens in a new tab)

   Recht, Roelofs, Schmidt & Shankar, 2019

   ICML. A freshly collected ImageNet test set cost models 11 to 14 points of accuracy.

5. [5]

   Calibrating probability with undersampling for unbalanced classification(opens in a new tab)

   Dal Pozzolo, Caelen, Johnson & Bontempi, 2015

   IEEE SSCI. Source of the widely used credit card fraud dataset: 492 frauds in 284,807 transactions.

6. [6]

   An introduction to ROC analysis(opens in a new tab)

   Fawcett, 2006

   Pattern Recognition Letters. The standard tutorial on ROC curves, AUC and its probabilistic meaning.

7. [7]

   The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets(opens in a new tab)

   Saito & Rehmsmeier, 2015

   PLOS ONE. Why ROC curves look reassuring on rare-event problems and PR curves do not.

8. [8]

   A study of cross-validation and bootstrap for accuracy estimation and model selection(opens in a new tab)

   Kohavi, 1995

   IJCAI. Empirical case for stratified 10-fold cross-validation.

9. [9]

   Leakage and the reproducibility crisis in machine-learning-based science(opens in a new tab)

   Kapoor & Narayanan, 2023

   Patterns. Documents leakage in 294 papers across 17 fields and proposes a taxonomy of eight leakage types.

10. [10]

    The Elements of Statistical Learning (2nd ed.), section 7.10.2: the wrong and right way to do cross-validation(opens in a new tab)

    Hastie, Tibshirani & Friedman, 2009

    Free textbook. The feature-selection leakage example reproduced in this lesson.

11. [11]

    On calibration of modern neural networks(opens in a new tab)

    Guo, Pleiss, Sun & Weinberger, 2017

    ICML. Deep networks became more accurate but less calibrated; temperature scaling fixes much of it.

12. [12]

    A careful examination of large language model performance on grade school arithmetic(opens in a new tab)

    Zhang, Da, Lee et al., 2024

    NeurIPS Datasets and Benchmarks. A fresh GSM8K-style test (GSM1k) showed accuracy drops of up to 8 points for some model families.

13. [13]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena(opens in a new tab)

    Zheng, Chiang, Sheng et al., 2023

    NeurIPS Datasets and Benchmarks. Strong LLM judges agreed with human preferences over 80% of the time, but showed position, verbosity and self-enhancement biases.

14. [14]

    The leaderboard illusion(opens in a new tab)

    Singh, Nan, Wang et al., 2025

    Shows how private testing of many variants and unequal data access can distort crowd-sourced arena rankings.