Confusion Matrix Explorer

About

Most classifiers do not output "yes" or "no". They output a score, and someone picks a threshold above which the answer counts as yes. That choice is not a technical detail: it decides who gets a follow-up test, which emails you never see, which card payments get blocked.

This lab gives you a set of scored examples and a threshold. Everything else, the confusion matrix, nine metrics, the ROC and precision-recall curves, their areas and the cost-optimal threshold, is computed exactly from those scores every time you move a slider.

Three things to notice as you explore:

• There is no threshold that improves everything. Lowering it buys recall with precision.
• The same model can be excellent or nearly useless depending on how common positives are. Drag prevalence down in the screening scenario and watch precision collapse while the ROC curve barely moves.
• The "best" threshold depends on what each mistake costs, which is a question about the world, not the model.

About the data. The scores are synthetic. Each scenario draws scores for 20,000 cases from two normal distributions (one per class), with a separation and spread chosen to resemble that kind of problem, then maps them into 0 to 1. The prevalences and costs are illustrative, not statistics about real screening programmes, spam or fraud. The metrics computed from them are real.

For the wider picture of evaluation, read the Model Evaluation lesson. To see whether scores can be read as probabilities, try the Calibration Lab.

How it works

From scores to a matrix

Every case has a true label and a score. At threshold t, cases with score at or above t are predicted positive. Crossing that with the truth gives four counts: true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). The histogram shows this directly: the yellow band is everything flagged; green area inside it is TP, blue area inside it is FP, and so on.

From a matrix to curves

Sweep t from 1 down to 0 and record, at every distinct score, the true positive rate TP / (TP + FN) and false positive rate FP / (FP + TN). That path is the ROC curve (Fawcett, 2006). Recording recall and precision instead gives the precision-recall curve. Your current threshold is the yellow dot on both.

The areas

AUC (area under the ROC curve) is computed here with the trapezoid rule over every distinct threshold. It has a clean meaning: the probability that a randomly chosen positive scores higher than a randomly chosen negative (Hanley and McNeil, 1982). 0.5 is a coin flip, 1 is perfect ranking. Because the scores come from a known model, the lab also shows the population AUC, Φ(d′ / √(1 + σ²)), so you can see sampling noise when positives are rare.

Average precision (AP) summarises the PR curve as the sum of precision at each threshold weighted by the gain in recall, the definition used by scikit-learn. Its floor is not 0.5 but the prevalence: a random ranker has precision equal to the share of positives.

Metrics

Every metric is a different ratio of the same four numbers. Which one matters depends on who pays for which mistake.

On imbalanced problems, MCC and the PR curve are usually more informative than accuracy and ROC (Chicco and Jurman, 2020) (Saito and Rehmsmeier, 2015).

Base rates

Recall and specificity are properties of the test: they are computed within each true class, so they do not depend on how many positives there are. Precision mixes the two classes, so it does. Bayes' rule makes the dependence exact:

precision = recall · p / (recall · p + (1 − specificity) · (1 − p))

where p is the prevalence. A worked example: a test with 90% recall and 91% specificity, used on a population where 1% have the condition. Out of 10,000 people, 100 are sick and the test catches 90. Of the 9,900 healthy people, 9% (891) also test positive. So 981 people test positive, and only 90 of them, about 9%, are actually sick. The test did not get worse; the population changed.

Try it: pick the screening scenario and drag prevalence from 50% down to 0.5%. The ROC curve and AUC stay put (up to noise as positives become few), while the PR curve sinks towards its new floor and the precision at your threshold falls. This is why a model evaluated on a balanced test set can disappoint in deployment, and why rare-event problems such as fraud should be judged on precision-recall, not ROC alone.

Costs and thresholds

If a false positive costs c_FP and a false negative costs c_FN, the best threshold is the one that minimises c_FP · FP + c_FN · FN. Turn on cost mode and the lab searches every distinct score for that minimum and draws the total cost at each threshold.

When the scores are calibrated probabilities, decision theory gives the answer in closed form: flag a case when its probability of being positive exceeds c_FP / (c_FP + c_FN) (Elkan, 2001). With a miss 20 times as costly as a false alarm, that is 1/21, under 5%. The scores in this lab are deliberately not calibrated (they are squashed normal scores), which is why the lab searches for the minimum numerically rather than using the formula. Real models are often miscalibrated too; the Calibration Lab shows how to check and fix that.

Things to try:

• Set the two costs equal on the two Gaussians scenario: the optimum sits near the point that maximises accuracy.
• In fraud, raise the cost of a miss and watch the optimum slide left into a sea of false alarms.
• Lower prevalence with costs fixed: the optimal threshold rises, because each flag is now less likely to be right.