AI for Decision Makers
Unit 11: Governance, Accountability, and Safety
The earlier units show a pattern: outcomes depend less on a model's properties than on the decision system around it — the label, the threshold, how people use the output, and whether anyone notices when it fails. Governance is the work of designing that system so that it performs, can be held to account, and fails safely. This unit covers the evidence on human oversight, the structures that assign accountability, the regulatory landscape as of late 2026, and the safety concerns that grow as systems gain autonomy.
Human-in-the-Loop: What the Evidence Says
'Keep a human in the loop' is the most common safeguard in AI policy, and the least examined. The evidence shows three distinct failure modes:
| Failure mode | What happens | Evidence |
|---|---|---|
| Automation bias | Reviewers defer to the system even when it is wrong | Systematic review in clinical decision support (Goddard et al. 2012) |
| Selective adherence | Reviewers follow the system when it agrees with them and override it otherwise, sometimes unequally across groups | Pretrial and sentencing risk assessment in Kentucky and Virginia (Stevenson 2018; Albright 2019; Stevenson & Doleac 2024) |
| Harmful overrides | Discretionary overrides make outcomes worse | Managers who overruled a hiring test hired workers who stayed for shorter periods (Hoffman, Kahn & Li 2018) |
Accountability Sinks and Moral Crumple Zones
In complex automated systems, responsibility tends to collapse onto the human operator nearest the failure, even when that person had little real control. Madeleine Clare Elish calls this a 'moral crumple zone' (2019). It protects the integrity of the technical system at the expense of the human. In institutions, this often takes the form of a frontline reviewer who signs off on hundreds of algorithmic recommendations a day and is then blamed for the ones that go wrong.
- Assign named accountability for the decision rule — the target, threshold, and objective (Unit 2) — not only for individual decisions.
- Separate the roles of system owner, vendor, operator, and independent reviewer, and write down what each is responsible for.
- Give affected people a route to contest decisions that ends with someone who has the authority to change the outcome.
Tools of Accountability
| Tool | What it does | Limitation |
|---|---|---|
| Impact assessment | Evaluates risks to rights and outcomes before deployment | Can become box-ticking; hard to anticipate effects in new contexts |
| Independent audit | Tests performance, disparities, and compliance | Requires access to models and sensitive data; standards still maturing |
| Interpretable models | For structured data, transparent models often match black-box accuracy and can be inspected directly (Rudin 2019) | Less available for images, text, and generative systems |
| Post-hoc explanations | Approximate why a black-box model produced an output | Can be unfaithful to what the model actually did; not a substitute for interpretability in high-stakes uses |
| Logging and monitoring | Records inputs, outputs, overrides, and outcomes over time | Useful only if someone reviews it and has authority to act |
The Regulatory Landscape (as of Late 2026)
The EU AI Act (Regulation (EU) 2024/1689) is the most comprehensive framework. Its obligations apply in phases, and the 2026 'Digital Omnibus' amendments (Regulation (EU) 2026/1744, in force since 27 July 2026) pushed back the high-risk deadlines:
| Obligation | Applies from |
|---|---|
| Prohibited practices (e.g. social scoring, certain manipulative systems) | 2 February 2025 |
| General-purpose AI model obligations | 2 August 2025 |
| Transparency duties for AI interactions and generated content (Article 50) | 2 August 2026 (marking duties for systems already on the market: 2 December 2026) |
| High-risk systems listed in Annex III | 2 December 2027 (originally 2 August 2026) |
| High-risk AI embedded in regulated products (Annex I) | 2 August 2028 (originally 2 August 2027) |
Dates reflect Regulation (EU) 2026/1744. Check the current consolidated text before relying on them.
Annex III maps closely onto this course. It covers AI used in education, employment and worker management, access to essential public and private services (including benefits eligibility and credit scoring), law enforcement, migration, and the administration of justice and democratic processes. Deployers of high-risk systems take on duties of their own: using systems as instructed, assigning competent human oversight, monitoring operation, and keeping logs. Public bodies, and certain private deployers, must also assess the impact on fundamental rights.
Outside the EU, regulation is mostly sectoral and varies by jurisdiction. Examples include medical-device rules (Unit 7), consumer-protection and anti-discrimination law, and state-level AI statutes in the US. Voluntary standards provide common structure: the NIST AI Risk Management Framework (2023) and its Generative AI Profile (NIST AI 600-1, 2024), and ISO/IEC 42001:2023 for AI management systems. National rules interact strategically (Unit 4), and divergence between jurisdictions is itself a risk to manage.
Safety: Risks That Grow With Capability
- Specification problems — systems optimize the objective they are given, not the one intended (Goodhart's Law: when a measure becomes a target, it ceases to be a good measure).
- Robustness — performance degrades under distribution shift, often silently (Units 1 and 3).
- Agency — as systems take actions through tools, errors compound and systems can be manipulated by content they read (Unit 12).
- Correlated failure — when many institutions rely on the same model or provider, one flaw becomes a system-wide failure (Unit 4's monoculture).
- Misuse — capable general-purpose systems lower the cost of harmful activities, from fraud to targeted harassment.
These concerns follow from Unit 1. If capability is growing and jagged, the risks that come with autonomy are a governance matter now, not only a speculative one. Proportionality still applies: match controls to what a system can do and the damage its errors can cause.
Monitoring After Deployment
Because of the engineering–science gap (Unit 1), pre-deployment testing cannot enumerate every failure. Governance therefore has to include continuous measurement after launch.
| Monitor | Why |
|---|---|
| Performance in your population, by group | Detects drift and emerging disparities |
| Base rates and input distributions | Early warning that the world has changed |
| Override and adherence rates, by reviewer and group | Detects selective adherence and rubber-stamping |
| Outcomes of the decisions, not just model metrics | The goal is better decisions, not better scores |
| Incidents and complaints | Captures failures that metrics miss |
Procurement as Governance
For most institutions, the contract is the most powerful governance instrument they have. Before signing, secure:
- Evidence of performance in a population like yours, with the right to validate locally before go-live.
- Documentation of training data, intended use, known limitations, and the outcome label the system predicts.
- Audit access for you or an independent party, including to logs and, where needed, to the model.
- Control over updates: notice, testing, approval, and rollback.
- Data rights and portability, so you can switch providers without losing your history or evaluations.
- Clear allocation of liability and incident-reporting duties.
Key Takeaways
- Human-in-the-loop fails in three ways: automation bias, selective adherence, and harmful overrides. Design and measure oversight.
- Assign accountability for the decision rule itself, and avoid moral crumple zones that blame operators for system design.
- Prefer interpretable models for high-stakes decisions on structured data; treat post-hoc explanations with caution.
- Know the EU AI Act timeline as amended in 2026: Annex III high-risk obligations now apply from 2 December 2027.
- Safety risks grow with autonomy and with shared dependence on a few providers; govern them proportionately.
- Monitor decisions and outcomes after deployment, and use procurement to secure validation, audit, update control, and exit.
Evidence & Further Reading
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127.
- Stevenson, M. (2018). Assessing Risk Assessment in Action. Minnesota Law Review, 103, 303.
- Albright, A. (2019). If You Give a Judge a Risk Score: Evidence from Kentucky Bail Decisions. Working paper.
- Stevenson, M. T., & Doleac, J. L. (2024). Algorithmic Risk Assessment in the Hands of Humans. American Economic Journal: Economic Policy, 16(4), 382–414.
- Hoffman, M., Kahn, L. B., & Li, D. (2018). Discretion in Hiring. Quarterly Journal of Economics, 133(2), 765–800.
- Elish, M. C. (2019). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging Science, Technology, and Society, 5, 40–60.
- Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206–215.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI).
- National Institute of Standards and Technology (2023). AI Risk Management Framework (AI RMF 1.0); (2024). Generative AI Profile (NIST AI 600-1).
- ISO/IEC 42001:2023. Information technology — Artificial intelligence — Management system.