Unit 10: Fairness, Bias, and the Impossibility Results

Unit 10 of 13

Fairness in algorithmic decisions is not a single property a system has or lacks. It is a set of competing definitions that formalize different moral intuitions. Some are mathematically incompatible under conditions that are common in practice. But the impossibility results are only one part of the picture. Measurement problems and comparisons with the human status quo matter at least as much, and they are where most practical decisions are won or lost.

Where Bias Enters

Problem formulation — what decision is being supported, and for whose benefit?
Measurement — does the label measure what we care about (offending vs. re-arrest, need vs. spending, learning vs. scores)?
Data — who is represented, and whose outcomes are missing (selective labels)?
Features — which inputs act as proxies for protected characteristics, legitimately or not?
Decision rule — which thresholds and error costs were chosen, and do they differ in effect across groups?
Use — who follows or overrides the system, and for whom (selective adherence)?
Feedback — do decisions shape the next round of data (feedback loops)?

Key Insight

Two stages are routinely overlooked: measurement and use. A model can be 'debiased' with respect to a biased label, and it can be statistically fair on paper and unfair in practice because people apply it selectively. Audit the decision system, not just the model.

Competing Definitions of Fairness

Definition	Formal requirement	Intuition it captures
Demographic parity	Equal rates of positive decisions across groups	Outcomes shouldn't differ by group
Equalized odds (error-rate balance)	Equal false positive and false negative rates across groups	People who behave the same way should face the same chance of error, whatever their group
Predictive parity	Equal positive predictive value across groups: a 'high-risk' flag is correct equally often	A flag should mean the same thing for everyone
Calibration within groups	For every score, the outcome rate is the same across groups	A score of 30% should mean 30% regardless of group

The Impossibility Results — Stated Precisely

Chouldechova showed that when groups have different base rates, a classifier with equal positive predictive value across groups cannot also have equal false positive and false negative rates — unless it predicts perfectly (2017). Kleinberg, Mullainathan, and Raghavan proved a related result for scores (2017). Calibration within groups and balance of average scores for the positive and negative classes cannot all hold together, except with equal base rates or perfect prediction.

Caution

The conditions matter. These definitions are not 'mutually exclusive' in general. They conflict when base rates differ and prediction is imperfect, which is the usual case in criminal justice, lending, and health. When they conflict, choosing a fairness criterion means choosing which kind of disparity you will accept, and that is a policy decision.

In the COMPAS dispute (Unit 5), ProPublica emphasized unequal error rates and Northpointe emphasized predictive parity. Given different re-arrest base rates, both findings were arithmetically consistent. The disagreement was about which property matters more.

When the Label Itself Is Biased

Every definition above is computed against an observed outcome. If that outcome is measured unequally — re-arrest reflects where police patrol, spending reflects access to care — then a system can satisfy any fairness metric with respect to the label while being unfair with respect to what the label was meant to measure. Obermeyer et al.'s health algorithm was well calibrated for cost. Its bias came entirely from the choice of cost as the label (Unit 7).

Key Insight

'Both sides were mathematically correct' holds only if the label is valid. Before debating which fairness metric to use, ask whether the outcome being measured is itself measured fairly. Often the highest-value fairness intervention is a better label.

Fairness Relative to the Status Quo

A system that falls short of an ideal fairness standard may still be fairer than what it replaces. Human decisions show measurable discrimination: Arnold, Dobbie, and Hull document it in bail decisions (2022). Hoffman, Kahn, and Li found that hiring managers who overruled a job test more often ended up with worse hires (2018). Kleinberg et al. estimate that algorithmic release rules could reduce racial disparities in pretrial detention while also reducing crime or jailing (2018). The opposite also happens. Selective use of risk scores widened some racial gaps in Kentucky, and in the Virginia courts that relied on scores most (Albright 2019; Stevenson & Doleac 2024).

The practical standard is comparative and empirical. Measure disparities in the current process, measure them under the proposed system as it will actually be used, and look for designs that improve on several dimensions at once. Such Pareto improvements are often available precisely because the status quo is so noisy.

The Costs of Fairness Interventions

Intervention	Benefit	Cost or risk
Improve the label	Removes bias at its source (e.g. need instead of spending)	Better labels can be costly to measure
Remove or constrain proxy features	Reduces indirect discrimination	Can reduce accuracy; other features may reconstruct the proxy
Group-specific thresholds	Can equalize error rates	May conflict with anti-discrimination law in some jurisdictions and domains; perceived as unequal treatment
Demographic parity constraints	Equal selection rates	May lower accuracy, and can increase errors within groups
Govern adherence and overrides	Prevents selective use from re-introducing bias	Requires logging, monitoring, and accountability for overrides
Independent audits	Detects problems and builds trust	Needs access to sensitive data and to the model; audit standards still maturing

These interventions are often worth their costs, but they are never free, and some interact with law. Decisions about them should be documented, owned, and revisited as evidence accumulates.

Key Takeaways

Fairness definitions formalize different intuitions; calibration within groups and predictive parity are distinct properties.
The impossibility results apply when base rates differ and prediction is imperfect — common conditions, but not universal ones.
Every fairness metric is computed against a label; a biased label can make a 'fair' system unfair. Fixing the label is often the best intervention.
Judge fairness comparatively: against the human status quo and against the system as it will actually be used, including overrides.
Fairness interventions have costs and legal dimensions. Choose them deliberately and assign ownership.

Evidence & Further Reading

Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163.
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017). Inherent Trade-Offs in the Fair Determination of Risk Scores. Proceedings of ITCS 2017.
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
Arnold, D., Dobbie, W., & Hull, P. (2022). Measuring Racial Discrimination in Bail Decisions. American Economic Review, 112(9), 2992–3038.
Hoffman, M., Kahn, L. B., & Li, D. (2018). Discretion in Hiring. Quarterly Journal of Economics, 133(2), 765–800.
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018). Human Decisions and Machine Predictions. Quarterly Journal of Economics, 133(1), 237–293.
Albright, A. (2019). If You Give a Judge a Risk Score: Evidence from Kentucky Bail Decisions. Working paper.
Stevenson, M. T., & Doleac, J. L. (2024). Algorithmic Risk Assessment in the Hands of Humans. American Economic Journal: Economic Policy, 16(4), 382–414.