Fairness Lab

About

What you are looking at

A bank scores loan applicants from two groups, A and B, and approves everyone at or above a threshold. Each dot is one of 200 applicants per group, placed by credit score. Green dots would repay, purple dots would default. Filled dots are approved, hollow ones are denied. Drag either chart, or use the sliders, to move a group's threshold.

Every approved loan that is repaid earns +300; every approved loan that defaults costs −700. So a loan is only worth making when the chance of repayment is above 0.7. That single number is the profit motive, and every fairness criterion you add pulls against it in a different direction.

How to use it

1. Start with Max profit. Note the two thresholds, and which fairness checks fail.
2. Try each strategy in turn. Watch the profit column and the checkmarks: every strategy satisfies its own criterion, and usually breaks the others.
3. Drag the Base-rate gap to zero and set group B's noise to 0.40. With identical groups every criterion can hold at once. Then raise it and watch that possibility vanish.
4. Raise Score noise for group B to model thin credit files. The score now means less for group B, and the profit-maximising bank reacts by demanding a higher score from them.

Why thresholds?

Almost every deployed risk model ends in a threshold: approve or deny, flag or ignore, admit or wait. The score may come from logistic regression or a large neural network, but the fairness questions are decided where the line is drawn. Hardt and colleagues showed that you can adjust thresholds after training to satisfy several fairness criteria without touching the model, which is what the strategy buttons do.

Fairness definitions

Each criterion is a statement about the confusion matrix of each group. Write TP for approved repayers, FP for approved defaulters, and so on. The lab shows every quantity live.

Both groups are approved at the same rate, regardless of who would repay.

In the lab: Compare the "Approved" figures under each strip.

Among people who would repay, the same share gets a loan in each group. Equal true positive rates.

In the lab: Compare "Repayers approved" (TPR). On the ROC chart the two markers sit at the same height.

Equal opportunity plus the same false positive rate: defaulters are also approved equally often in both groups.

In the lab: On the ROC chart the two markers land on the same point.

An approval means the same thing in both groups: the same share of approved applicants repay. Equal precision.

In the lab: Compare "Precision". This is the thresholded cousin of calibration.

Why equalized odds needs a coin flip

A single threshold can only put a group somewhere on its ROC curve. When the two groups' curves differ, they typically cross at no useful point, so no pair of plain thresholds gives both groups the same true and false positive rates. Hardt, Price and Srebro's fix is to randomise: approve everyone above an upper threshold, and approve people between a lower and upper threshold with some probability. Mixing two thresholds reaches any point on the straight line between them on the ROC chart, which lets both groups land on a common point.

The Equalized odds button finds the most profitable such point (it lies in the overlap of the two groups' achievable regions) and shows the random band as a hatched strip with its probability. The metrics are then expected values. Whether a lender or court could accept two identical applicants getting different answers by coin flip is a real objection, and part of why equalized odds is often discussed but rarely deployed in this form.

Where calibration fits

A score is calibrated within groups if, for each group, applicants with a given score repay at the rate the score implies. In this simulation the score is a noisy reading of the same underlying quantity for both groups, so a score of 60 means slightly different things for A and B whenever the groups differ; raise the noise slider and the difference grows. Predictive parity is the version of calibration that survives thresholding: among approved people, the repay rate is the same.

The trade-off

The impossibility results

In 2016 and 2017 two papers proved, independently, that the argument you can reproduce here is not a quirk of one dataset. Kleinberg, Mullainathan and Raghavan(opens in a new tab) showed that a risk score cannot simultaneously be calibrated within groups and have balanced error rates for both classes across groups, except when the groups have equal base rates or the predictor is perfect. Chouldechova(opens in a new tab) gave the thresholded version as one line of algebra:

FPR = p / (1 − p) × (1 − PPV) / PPV × (1 − FNR)

where p is the base rate of the outcome being predicted, PPV is precision and FNR is the false negative rate. If two groups have different p and you insist on equal PPV (predictive parity) and equal FNR (equal opportunity), the formula forces their false positive rates apart. Something has to give.

What this lab adds: the price of each choice

The strategy list reports the profit of each option next to the criteria it satisfies. Constraints cost money here because the bank's objective ignores who is harmed by errors. That is a modelling choice, not a law of nature: a bank that also counted the social cost of wrongly denying creditworthy applicants would draw different lines.

So which criterion is right?

None universally. The question is which error hurts whom. In lending, a false negative denies credit to someone who would have repaid, so equal opportunity targets the harm to applicants. In pretrial risk assessment, false positives detain people who would not have reoffended, which is what ProPublica's COMPAS analysis measured, while the vendor defended the tool on predictive parity. Both were right about their own metric. Choosing between them is a policy decision that should be made openly, by people accountable for it, and not buried in a threshold.

Why "just remove the group label" does not work

The Group-blind strategy never looks at group membership, yet when base rates differ it typically fails most of the criteria. And in real data, dropping the protected attribute rarely even makes a model blind: postcode, school, shopping patterns and word choice carry the same information. Blindness is a statement about inputs; fairness criteria are statements about outcomes.

Real cases

Four documented cases, each showing a different way bias enters a system. The lab's toy world only models the last step, the threshold; these show why the earlier steps matter just as much.

COMPAS recidivism scores

ProPublica analysed risk scores for defendants in Broward County, Florida. Among defendants who did not reoffend within two years, 44.9% of Black defendants had been labelled higher risk, against 23.5% of white defendants. Among those who did reoffend, 47.7% of white defendants had been labelled low risk, against 28.0% of Black defendants.

Unequal false positive and false negative rates, while the vendor pointed to similar predictive value across groups: the impossibility result in the wild.

Larson, Mattu, Kirchner and Angwin, ProPublica (2016)(opens in a new tab)

Gender Shades

Buolamwini and Gebru tested three commercial gender classifiers on a benchmark balanced by skin type and gender. Error rates reached 34.7% for darker-skinned women, while the maximum for lighter-skinned men was 0.8%.

Aggregate accuracy hid a 40-fold gap. Always break metrics down by group, and by intersections of groups.

Buolamwini and Gebru, FAT* (2018)(opens in a new tab)

Amazon recruiting model

Reuters reported that an experimental résumé-screening model, trained on a decade of past applications, learned to penalise résumés containing the word "women's" and downgraded graduates of two all-women's colleges. Amazon scrapped the project.

Historical labels encode historical decisions. Removing gender as an input did not remove the proxies for it.

Dastin, Reuters (2018)(opens in a new tab)

Healthcare risk algorithm

Obermeyer and colleagues found that a widely used algorithm predicted future healthcare costs as a stand-in for health needs. Because less money had historically been spent on Black patients with the same level of need, Black patients were sicker than white patients at the same score. Correcting it would raise the share of Black patients flagged for extra care from 17.7% to 46.5%.

The target variable itself was the bias. A perfectly accurate cost predictor was still the wrong tool.

Obermeyer, Powers, Vogeli and Mullainathan, Science (2019)(opens in a new tab)