AI for Decision Makers
Unit 7: Healthcare
Healthcare shows AI at its most promising and its most overhyped, often in the same year. The relevant baseline is a system with real limits: diagnostic error is a recognized patient-safety problem, specialists are scarce in many places, and clinicians spend large parts of their day on documentation. The question is never 'AI or a perfect doctor?' It is which tasks, validated how, inside which workflow, and compared with what patients actually receive today.
The Institutional Landscape
| Application | What is predicted or produced | What remains a judgment |
|---|---|---|
| Screening & imaging | Probability that an image shows disease | Referral, follow-up, and how to communicate uncertainty |
| Deterioration alerts | Risk of sepsis or decline in the next hours | Whether to act on an alert, given alarm fatigue and competing demands |
| Care-management targeting | Which patients will need the most care | Who should receive scarce programme places, and on what basis |
| Treatment recommendation | Options ranked for a patient profile | Whether evidence applies to this patient, and their preferences |
| Clinical documentation | Draft notes from recorded conversations | Accuracy, omissions, and what belongs in the record |
Decision: Diagnosis
Medical Diagnosis Through the Five Stages
| Stage | Traditional | AI-Mediated | Risk |
|---|---|---|---|
| 🔍Framing | Clinician forms a differential diagnosis; quality varies with experience and time pressure | Task is defined by the model's input (one imaging modality, a fixed set of findings) | Conditions outside the model's scope are invisible to it |
| 📊Prediction | Clinician estimates likelihood; diagnostic errors and variation are well documented | Model estimates probability from patterns in labelled cases | Performance can drop at new sites, on new equipment, or for under-represented patients |
| ⚖️Decision rule | Clinician weighs harms of testing or treatment against waiting, with the patient | Alert thresholds and referral cut-offs encode the trade-off | Thresholds set for one population or workload may produce floods of false alarms in another |
| 🎯Action | Tests, treatment, referral, or reassurance | Alerts, worklist prioritization, or autonomous screening decisions | Alarm fatigue leads staff to ignore alerts, including correct ones |
| 🔄Feedback & adaptation | Follow-up is incomplete; missed diagnoses are often never learned about | Outcomes feed monitoring and retraining | Verification bias: true status is confirmed mainly for patients who were tested, so errors on untested patients stay hidden |
Where AI Expands Access
In April 2018, the FDA authorized IDx-DR through its De Novo pathway. It was the first device permitted to give a diabetic-retinopathy screening decision in primary care without a clinician interpreting the image. In its pivotal study of 900 patients at 10 primary-care sites, it achieved 87.4% sensitivity and 89.5% specificity for more-than-mild retinopathy. Many people with diabetes miss recommended eye exams, so the relevant comparison is often not an ophthalmologist but no screening at all. Here the opportunity cost of not adopting (Unit 4) is undiagnosed disease.
Case Study: IBM Watson for Oncology
Watson for Oncology — Marketing Ahead of Evidence
Watson for Oncology was marketed internationally as a system that would bring elite cancer expertise to hospitals everywhere. Internal IBM documents reported by STAT in 2018 described 'multiple examples of unsafe and incorrect treatment recommendations'. The documents traced the problems largely to how the system was trained: on a small number of synthetic (hypothetical) cases prepared by a handful of specialists at Memorial Sloan Kettering, reflecting their preferences rather than outcome evidence. No patient harm was reported.
Watson for Oncology is sometimes taught as an example of AI that 'worked in the lab but failed to generalize'. The record suggests something more mundane and more useful for decision-makers. There was little evidence it worked in the first place, and the evidence it lacked was the kind any purchaser could have requested.
Lessons
- A system trained to reproduce one institution's preferences is not a system that learned what works for patients
- Absence of independent, published, prospective validation should have been disqualifying for a clinical recommendation tool
- Commercial incentives to scale ahead of evidence are a predictable principal–agent problem (Unit 4)
- Not every AI failure is mysterious: this one had well-understood causes that ordinary due diligence could have caught
Validation in the Wild: The Epic Sepsis Model
Unit 3 introduced the independent validation of the Epic Sepsis Model. It was widely deployed, but at Michigan Medicine it discriminated poorly (AUC 0.63), missed about two-thirds of sepsis cases at the recommended threshold, and alerted on 18% of hospitalized patients (Wong et al. 2021). The operational lesson goes beyond accuracy. An alert that is usually wrong teaches clinicians to ignore alerts, which degrades the safety value of every alert, including correct ones. Evaluate an alerting system by what clinicians do with it and by patient outcomes, not only by its score on a validation set.
The Label Problem: Obermeyer et al.
A widely used algorithm selected patients for 'high-risk care management' programmes: extra nursing and coordination for people expected to have complex needs. It predicted future healthcare costs, a reasonable-sounding proxy for need. But less money is spent on Black patients with the same level of illness, so at any given risk score Black patients were considerably sicker than white patients (Obermeyer et al. 2019). The algorithm predicted cost accurately. The problem was the choice of what to predict.
Obermeyer and colleagues estimated that if the algorithm targeted health need rather than cost, the share of Black patients automatically identified for the programme would rise from 17.7% to 46.5%. Working with the developer, they showed that changing the label sharply reduced the bias. The case is encouraging as well as cautionary: label bias, once found, can be fixed by changing what the model predicts.
Regulation: FDA Pathways and Adaptive Software
| FDA pathway | When it applies | AI example | Key requirement |
|---|---|---|---|
| 510(k) clearance | Low-to-moderate risk with a legally marketed predicate | Most AI imaging aids and triage tools | Substantial equivalence to the predicate device |
| De Novo classification | Novel low-to-moderate risk with no predicate | IDx-DR autonomous retinopathy screening (2018) | Reasonable assurance of safety and effectiveness; creates a new device type |
| Premarket approval (PMA) | High-risk (Class III) devices | Few AI devices to date | Extensive clinical evidence of safety and effectiveness |
Traditional device regulation assumes a product that doesn't change after authorization. Learning systems do. In December 2024 the FDA issued final guidance on Predetermined Change Control Plans (PCCPs) for AI-enabled devices. A manufacturer can obtain authorization in advance for specified kinds of modification, such as retraining on new data, together with a protocol for data management, performance evaluation, and monitoring. For a deploying institution, a PCCP shifts the key question to who verifies that each update performs in your population.
Allocation Under Scarcity: Values Before Algorithms
During COVID-19, many crisis standards of care ranked patients for ventilators and ICU beds using clinical scoring rules such as the SOFA score — simple point systems, not machine learning. The value question came first: maximize lives saved, life-years, need, or use a lottery among similar patients? An analysis of 113,158 patients found that SOFA overestimated mortality for Black patients. In simulated allocation, 9.4% of Black patients would have been wrongly excluded from the highest priority group (Ashana et al. 2021).
Generative AI in the Clinic
The fastest-spreading clinical AI is not diagnostic but administrative: ambient scribes that draft notes from recorded patient conversations. In an early deployment at The Permanente Medical Group, 3,442 physicians used ambient scribes in over 300,000 encounters in the first ten weeks. The evaluation reported less time spent on documentation and positive physician feedback, with notes still requiring clinician editing (Tierney et al. 2024). Lower-risk workflow uses like this can deliver real value early. They still need governance: patient consent, checks for omissions and fabricated details, and clarity that the signing clinician is responsible for the record.
Key Takeaways
- Compare AI with the care patients actually receive. Where specialists are scarce, the alternative may be no screening at all.
- Watson for Oncology failed for well-understood reasons — synthetic training cases and no independent validation — that due diligence could have caught.
- Evaluate alerting systems by clinical behaviour and outcomes; frequent false alarms erode the value of every alert.
- Label choice drives equity: Obermeyer et al. show an accurate model can encode unequal access, and that changing the label can fix it.
- Know the FDA pathway behind a product, and for adaptive systems, who validates each update locally.
- Allocation rules embody value choices that predate AI; check any scoring rule — AI or not — for calibration across groups.
Evidence & Further Reading
- U.S. Food and Drug Administration (2018). FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems (IDx-DR, De Novo authorization), April 11, 2018.
- Ross, C., & Swetlitz, I. (2018). IBM's Watson supercomputer recommended 'unsafe and incorrect' cancer treatments, internal documents show. STAT, July 25, 2018.
- Wong, A., Otles, E., Donnelly, J. P., et al. (2021). External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine, 181(8), 1065–1070.
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
- U.S. Food and Drug Administration (2024). Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. Final guidance, December 2024.
- Ashana, D. C., et al. (2021). Equitably Allocating Resources during Crises: Racial Differences in Mortality Prediction Models. American Journal of Respiratory and Critical Care Medicine, 204(2), 178–186.
- Tierney, A. A., et al. (2024). Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical Documentation. NEJM Catalyst Innovations in Care Delivery.