Calibration Lab

About

A classifier that says 0.8 is making a promise: of all the cases where it says 0.8, about 80% should turn out positive. When that promise holds the model is calibrated, and software can use its probabilities directly: to set thresholds, to weigh costs, or to decide when to hand a case to a person.

Modern neural networks often break the promise. Guo et al. (2017) showed that deep image classifiers had become markedly overconfident compared with older, smaller networks, and that a one-parameter fix, temperature scaling, repaired most of it. This lab lets you reproduce that story in miniature: train a small network too long, watch its reliability diagram bend away from the diagonal, then fix it.

Calibration is also why decision models such as TypeSafe's Jev advertise calibrated probabilities rather than just labels. The Decision Models lesson covers that side.

How it works

The model. A one-hidden-layer tanh network (or plain logistic regression when you pick Linear) is trained full-batch with Adam on binary cross-entropy. Three splits are drawn from the same distribution: training points (shown on the map), 500 validation points used only to fit the calibrator, and 1,500 test points used for every number and chart.

Reliability diagram. Test predictions are sorted into equal-width bins by predicted probability. For each bin we plot the mean prediction against the observed fraction of positives. Points below the diagonal mean the model was too sure of "yes"; points above it mean too sure of "no".

Expected calibration error. ECE = Σ_b (n_b/N) · |mean prediction_b − observed frequency_b|, the count-weighted average of the red gaps. It depends on the number of bins, which is why the bins slider is there: too few bins hide miscalibration, too many make each bin noisy.

Brier score and log loss are proper scoring rules: they are minimised only by the true probabilities, and they reward both calibration and sharpness. A model that always predicts the base rate is perfectly calibrated but useless, and these scores notice.

Fixing it

Divide every logit by one number T, chosen to minimise log loss on validation data (here with a golden-section search). T > 1 softens an overconfident model. Because it only rescales, the ranking of cases and the accuracy never change.

Fit p = sigmoid(a·z + b) on validation data with Newton’s method, using Platt’s smoothed targets. The extra bias term b can also correct a model that leans toward one class, at the cost of one more parameter.

Fit any non-decreasing step function from score to probability with pool-adjacent-violators. Very flexible, so it needs more validation data and can overfit, which often shows up as a worse log loss even when ECE improves.

Deciding and deferring

A system does not have to answer every case. In selective prediction it acts only when its confidence clears a threshold and passes the rest to a person or a slower model. The coverage curve shows the trade: as you handle fewer cases automatically, accuracy on the ones you keep goes up.

Calibration is what makes the threshold honest. With an overconfident model, "act when confidence is at least 0.9" might deliver 85% accuracy. After calibration the promised and delivered columns line up, so a product owner can pick a threshold from a target error rate instead of by trial and error.

Notice that temperature scaling does not change which cases are most confident, so the curve itself barely moves. What changes is the meaning of the numbers along it. This is the same pattern TypeSafe's documentation recommends for Jev: act on high confidence, confirm medium, route low confidence to a human, and set stricter thresholds for riskier actions.

Try this

1. Press Train and let it run past 2,000 epochs. Test accuracy barely changes, but ECE and log loss climb. Where does the histogram pile up?
2. Now pick Temperature. How large is the fitted T? Check that accuracy is identical before and after.
3. Reset, set weight decay to 0.01 and train again. Regularisation keeps the network calibrated for much longer.
4. Switch to Linear. Logistic regression on blobs is close to calibrated out of the box, as Niculescu-Mizil and Caruana (2005) found.
5. Switch to Synthetic scores and set bias to +1.5. Temperature scaling cannot fix a shift; Platt scaling can.
6. Set the deferral threshold to 0.95 on an overtrained raw model. Compare promised and delivered accuracy, then calibrate.