AI for Decision Makers
Unit 3: The Decision-Maker's Quantitative Toolkit
You don't need to build models to govern them, but you do need a small amount of arithmetic. Most costly AI decisions in this course could have been avoided by a leader asking a handful of quantitative questions. This unit gives you those questions, with worked examples. Every later unit uses them.
1. Base Rates: Why Most Alerts Can Be False
A vendor says its fraud model catches 90% of fraud with a 1% false-positive rate. That sounds excellent. Now apply it to a realistic base rate: 1,000,000 transactions, of which 0.2% (2,000) are fraudulent.
| Flagged | Not flagged | Total | |
|---|---|---|---|
| Fraudulent | 1,800 (90% caught) | 200 (missed) | 2,000 |
| Legitimate | 9,980 (1% false positives) | 988,020 | 998,000 |
| Total | 11,780 | 988,220 | 1,000,000 |
Worked example (illustrative numbers)
Only 1,800 of 11,780 flags — about 15% — are actually fraud. That figure is the positive predictive value (PPV). The other 85% of flagged customers are innocent, and your staff will spend most of their review time on them. Nothing about the model changed; the rarity of the event did all the work. Real deployments show the same pattern. In an independent validation of a widely used hospital sepsis alert, only about 12% of alerted patients actually had sepsis (Wong et al. 2021).
2. Calibration: Does 30% Mean 30%?
A model is calibrated if, among all the cases it scores at 30%, about 30% actually have the outcome. Calibration is what lets you treat a score as a probability and reason about costs. Ask for a calibration plot from your own population, broken down by the groups you serve. Two warnings apply. First, a model can be well calibrated and still poor at separating cases (everyone scored near the average). Second, calibration often drifts when a model moves to a new setting, or when the world changes after training.
A common summary metric, the AUC, is the probability that a randomly chosen positive case is scored above a randomly chosen negative one. It is useful for comparing models, but it says nothing about performance at the threshold you will actually use, and nothing about calibration.
3. Thresholds Are Where Judgment Lives
A calibrated probability does not tell you what to do. A decision rule does. If acting on a case that turns out negative costs C(false alarm), and failing to act on a case that turns out positive costs C(miss), the expected-cost-minimizing rule is: act when the probability exceeds C(false alarm) ÷ (C(false alarm) + C(miss)).
| If a miss costs… | …act when probability exceeds | Example setting |
|---|---|---|
| the same as a false alarm | 50% | Routine prioritization where both errors are cheap |
| 4× a false alarm | 20% | Follow-up call for a patient at risk of readmission |
| 9× a false alarm | 10% | Screening for a serious, treatable condition |
| ¼ of a false alarm | 80% | Actions that are costly or coercive for the person flagged, such as detention or account closure |
The cost ratio is a value judgment. Choosing it is exactly the judgment that Unit 2 says relocates upstream. It is usually set implicitly, by a default threshold in a vendor's software. Make it explicit, document who chose it and why, and revisit it. Two institutions with the same model but different cost ratios are running different policies.
4. Count the Decisions That Change
A system only creates value where it changes what you would otherwise have done. If experienced staff and the model agree on 80% of cases, the model's accuracy on those cases is irrelevant. Its value — and its risk — lies entirely in the 20% where they disagree. Evaluate a proposal by asking how many decisions it will change, in which direction, and whether those changes are improvements. That is the economist's marginal reasoning applied to decisions (Unit 4).
Beware selective labels. You only observe outcomes for the actions you took: you see whether released defendants returned to court, never whether detained ones would have. A model trained and evaluated only on the cases a previous policy let through can look better or worse than it really is (Kleinberg et al. 2018). Credible evaluation needs a design that accounts for this — often a randomized or staggered rollout.
5. Risk Is Not the Same as Benefit
Many institutional uses of prediction are really about targeting an intervention: whom to give extra care, mentoring, inspection, or outreach. The natural move is to target the highest-risk cases. But the decision-relevant quantity is the benefit of the intervention to each person: their outcome with it minus their outcome without it. Risk and benefit can point in opposite directions (Athey 2017).
| Group (100 students each) | Dropout risk without mentoring | Dropout risk with mentoring | Dropouts prevented per 100 places |
|---|---|---|---|
| A — highest predicted risk | 60% | 55% | 5 |
| B — moderate predicted risk | 30% | 15% | 15 |
Hypothetical illustration: targeting by risk picks Group A; targeting by benefit picks Group B and prevents three times as many dropouts
A risk model cannot tell you which group benefits more. That requires evidence about the intervention's effect, ideally from randomized evaluation, and models designed to estimate effects rather than risk. If a proposal uses a risk score to allocate a programme, ask what evidence shows that the highest-risk people benefit most.
6. Always Compare With a Simple Baseline
Complexity and secrecy are not evidence of accuracy. When researchers tested the commercial COMPAS recidivism tool on Broward County data, it achieved 65.2% accuracy. A logistic regression using only two inputs — age and number of prior convictions — achieved 66.8%, and the pooled predictions of untrained online volunteers reached 67.0% (Dressel & Farid 2018). For high-stakes decisions on structured data, transparent models often match black-box performance (Rudin 2019).
7. External Validation: Performance Travels Badly
The Epic Sepsis Model was deployed in hundreds of US hospitals. Researchers at Michigan Medicine evaluated it independently on 38,455 hospitalizations. They found an AUC of 0.63, well below the performance reported by the developer. At the recommended threshold it identified only about a third of patients who developed sepsis, while alerting on 18% of all hospitalized patients (Wong et al. 2021). The developer disputed the study's definition of sepsis onset. That is itself a lesson: performance depends on how the outcome is defined and measured locally.
- Ask for performance measured on data from your institution or one like it, not only the developer's.
- Prefer prospective evaluation (scoring real cases going forward) to retrospective replays.
- Ask for results broken down by the groups you serve and by the conditions you operate in.
- Ask who did the evaluation and whether they were independent of the vendor.
Reading a Vendor Claim
| The claim | The question to ask |
|---|---|
| "95% accurate" | At what base rate? What share of our flags will be correct (PPV), and how many cases will we miss? |
| "AUC of 0.90" | What are sensitivity and PPV at the threshold we would actually use? Is it calibrated in our population? |
| "Clinically / externally validated" | Where, on whom, prospectively or retrospectively, and by whom? |
| "Reduces bias" | By which fairness measure, relative to which baseline, and measured against which outcome label? |
| "Outperforms experts" | Which experts, on what task, with what information — and does it improve decisions when experts use it in practice? |
| "Learns continuously" | How is drift detected, who approves updates, and can we roll back? |
| "Proprietary model" | What does a simple transparent model achieve on the same data? What audit access will we have? |
Key Takeaways
- Translate every accuracy claim into consequences at your base rate: share of flags that are correct, cases missed, review workload.
- Demand calibration in your population; treat AUC as a comparison tool, not a performance guarantee.
- Thresholds encode value judgments about error costs. Set them deliberately and assign an owner.
- A system's value lies in the decisions it changes. Evaluate those, and design evaluations that handle selective labels.
- Targeting by risk is not targeting by benefit; allocating interventions needs evidence about effects.
- Insist on a simple-baseline comparison and on independent, local, prospective validation.
Evidence & Further Reading
- Wong, A., Otles, E., Donnelly, J. P., et al. (2021). External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine, 181(8), 1065–1070.
- Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1), eaao5580.
- Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206–215.
- Athey, S. (2017). Beyond prediction: Using big data for policy problems. Science, 355(6324), 483–485.
- Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018). Human Decisions and Machine Predictions. Quarterly Journal of Economics, 133(1), 237–293.