Who Designed the Loop?

The comfortable phrase

"Human in the loop" is a comfortable phrase. I heard it declared several times at WSDC today, and each time people nodded. No one asked what the human actually does.

In most deployed systems I see, the human is a checkpoint at the edge of the loop, not a participant in it. They approve outputs at volume, or review a sample, or get alerted after the fact. Approving at that scale isn't oversight. It's a signature. And as the technology improves, that role will shrink, because the capability to do the work without them already exists.

None of this means humans are out. It means the question has to change. The honest question isn't "is there a human in the loop?" It's "where do humans matter most, and are we putting them there?"

What the evidence says

The evidence that review-style oversight often fails is strong, and it has been accumulating for years. Four findings stand out.

People follow the machine when it is wrong. In a 2023 study, 27 radiologists each read 50 mammograms with a purported AI assistant. When the AI's suggestion was correct, accuracy was about 80% at every experience level. When it was wrong, accuracy fell to roughly 20% for inexperienced readers, 25% for moderately experienced readers, and 45% for the most experienced. Experience reduced the effect but did not remove it. This was a lab experiment, and it did not compare readers with and without AI, so it shows the mechanism rather than a real-world error rate. (AuntMinnie summary)

Human plus AI is not reliably better than the best of either. A 2024 meta-analysis in Nature Human Behaviour pooled 106 experiments and 370 effect sizes. On average, human-AI combinations did worse than the better of human or AI alone. Where the human was the stronger party, the combination gained. Where the AI was stronger, it lost. The losses clustered in decision tasks, while content creation showed gains. A later re-analysis found that studies giving people feedback on outcomes showed relatively higher synergy. (Vaccaro et al.; re-analysis)

Approval habits drift toward automation. Anthropic's study of millions of interactions across Claude Code and its API found about 20% of sessions among new users used full auto-approve, rising to over 40% by 750 sessions. Experienced users also interrupted more often, and Anthropic reads this as oversight changing form, from approving each action to monitoring. Anthropic is a vendor and the study covers one product, whose default settings may shape the pattern. (Anthropic)

Policy research reaches the conclusion. Ben Green surveyed 41 policies requiring human oversight of government algorithms. He found that people cannot reliably perform the oversight functions the policies assume, and that the policies then lend legitimacy to flawed systems and leave frontline operators holding the blame. This is sometimes called the moral crumple zone. His proposed answer is institutional oversight in place of individual oversight. (Green; Masood)

The regulatory trap

Europe's AI Act requires human oversight of high-risk systems, and it assumes that oversight works. Melanie Fink's analysis of Article 14 notes that the empirical evidence shows significant limits to human oversight, and warns against treating it as a standalone safeguard. A policy brief from the ZEW research institute went further, arguing that Article 14 should acknowledge how hard it is for people to assess algorithmic recommendations, and that oversight should be tested empirically for each high-risk use. The European Data Protection Supervisor has said oversight must be more than a tick-box exercise. (Fink; ZEW; EDPS summary)

The timeline has also moved. The Digital Omnibus, Regulation (EU) 2026/1744, entered into force on 27 July 2026. It pushed standalone high-risk obligations to 2 December 2027 and product-embedded systems to 2 August 2028. The obligations were postponed, not removed. (BigID; analysis)

That delay is a window. What gets built into systems during it is what regulators will later ask humans to oversee.

The loop with no foundation

I have seen a lot of loops with no foundation this year, and it worries me. Review steps, escalation paths and audit trails, all wrapped around a system whose underlying assumptions no one examined. The loop looks rigorous. It is standing on assumptions no one tested.

Here is a pattern I keep hearing. A senior leader at a pharmaceutical company told me they are very careful about the data they use to train their systems: curated data only. That care is real, and it matters. What I did not hear was anything about the system underneath: what it is told to look for, which edges it should care about, which assumptions it must never make, and who decided. These are capable people with big jobs, and the gap is easy to miss precisely because the controls they have built are genuine.

Curated data is checkable. You can inspect a dataset and trace it to its source. A definition of quality usually is not, because it lives in a prompt, in an unstated sense of what "good" means, or nowhere at all. So errors in the definition surface late, and they look like model failures rather than specification failures.

Measurement science has a name for the underlying problem: construct validity, meaning whether a measure captures what it claims to. A 2026 paper on AI benchmarking notes that validity has received limited attention in AI evaluation, and earlier work by Jacobs and Wallach showed that fairness metrics routinely measure constructs different from their stated targets. Practitioner interviews on data quality in AI-driven systems found that using models as quality assessors introduced circularity: the system helps define the standard it is then judged against. A theoretical analysis of clinical AI makes a related point, that distortion built into health records is a problem of meaning, not only of data quality in the conventional sense. (benchmark validity; construct validity in agentic evaluation; practitioner interviews; clinical AI)

Not a new idea, an ignored one

The argument that human judgment matters most upstream has a long history. It has not crossed into most enterprise AI conversations.

In philosophy, Filippo Santoni de Sio and Jeroen van den Hoven developed a theory of meaningful human control in the debate over autonomous weapons, starting from the point that a human in the loop is not enough. It rests on two conditions. Tracking: the system responds to the relevant human moral reasons. Tracing: the system's behavior can be traced to a proper moral and technical understanding on the part of at least one human who designs or interacts with it. A separate interdisciplinary analysis argues that effective oversight requires the overseer to have causal power, epistemic access, self-control and fitting intentions, and treats the technical design of the system as one of the factors. (Santoni de Sio and van den Hoven; tracking and tracing; effectiveness analysis)

The defense literature applies this more concretely than anything I found in the civilian world. A West Point Modern War Institute analysis argues that late-stage human-in-the-loop overrides are the least trustworthy way to fix errors arising from an algorithmic system. Upstream, it says, human judgment sets parameters, calibrates trade-offs and applies doctrinal and ethical constraints, while design-level control and operational monitoring coexist. A Just Security piece makes a parallel point: human control starts with the code, with judgment embedded through data selection, machine-learning choices, testing and interface design, and at command level through mission planning and rules of engagement. A 2018 UK position on autonomous weapons was described by a defense journalist as almost a "human-in-the-code" model. (West Point MWI; Just Security; C4ISRNET)

Having the concept does not guarantee applying it. A Small Wars Journal essay asks whether meaningful control was exercised across the methodology that produced a targeting recommendation, not only at the approval step, and warns that commanders can keep legal authority while the judgment behind a recommendation erodes. (Small Wars Journal)

Outside AI and defense, safety engineering reached a similar conclusion decades ago. In the STAMP safety framework from MIT, software-related accidents usually trace to flawed requirements rather than coding errors. The Mars Polar Lander is the standard case: no component failed, and every software requirement was met, but the requirements were inadequate. In machine learning, Sambasivan and colleagues interviewed 53 practitioners and found that data cascades, compounding downstream failures from data problems, were pervasive and often avoidable, and that they tend to originate early, at the stage of data definition. Neither body of work is about ontology as such. Both are about what gets specified before the system runs. (Leveson; Thomas, MIT; Sambasivan et al.; Google Research)

A note on the word ontology, because it is used three ways. In philosophy, an ontology is a system of categories, an account of what kinds of things there are. A theory's ontological commitments are the things it must treat as existing for it to be true. In knowledge engineering, Tom Gruber defined an ontology as an explicit specification of a conceptualization, and observed that for an AI system, what exists is what can be represented. In much enterprise marketing, the word now means any shared schema of entities and relationships. I mean the first sense, made explicit where possible in the second. Gruber's point matters most here: every knowledge-based system is committed to some conceptualization, explicitly or implicitly, so a system with no examined ontology still has one. It simply was not chosen. The Stanford Encyclopedia adds that leaving out categories a domain needs, such as time or action, is itself an error of omission. (Stanford Encyclopedia: Ontological Commitment; Stanford Encyclopedia: Ontology and Information Systems; Gruber; Guarino, Oberle and Staab)

On the enterprise side, ontologies and semantic layers are now marketed as a control plane between human intent and AI execution. Some of that uses the word in the knowledge-engineering sense and some in the looser schema sense, and much of it is vendor material, which shows the idea is circulating, not that it works. It rarely engages the oversight critique above. In the sources I reviewed, the two conversations barely cite each other. We need more cross-domain willingness. (example whitepaper)

Both layers

The design layer and runtime oversight are not alternatives. The design layer decides what the runtime layer can see and catch.

Design alone is not enough. As Fink reads it, Article 14 requires oversight during use, not only at design, and the West Point analysis insists the two coexist. A system with a careful ontology still drifts, meets cases no one anticipated, and gets used in ways its designers did not foresee. (AI Governance Library on Fink)

The design layer is every decision made before the first output: what the system may represent, what counts as a valid inference, what counts as evidence and quality, and who owns those definitions and can revise them. The runtime layer is oversight that is measured instead of assumed. One practical test is to seed known wrong outputs into live review queues and make the catch rate the primary oversight metric. Another is to track how long it takes to detect a problem and to intervene. (Masood; Arize)

Small experiments suggest this layer changes system behavior. In one, free-text tool parameters produced hallucinated identifiers in 43% of 72 queries, against none when the parameters were constrained to a domain schema. In another, adding an ontology-based query check cut a question-answering system's error rate from 83.3% without a knowledge graph to 19.44%. These are preprints on narrow tasks, and the first uses a schema far thinner than an ontology in the philosophical sense. They show that specification changes errors, not that it improves human oversight. (tool-call study; question-answering study)

Three limits apply to this argument, and I would rather name them than have a reader find them.

1. Design-time definitions are never finished. A paper on reasons underdetermination in meaningful human control argues that the relevant reasons cannot be fully specified in advance, which challenges both tracking and tracing. An ontology therefore needs an owner, versioning and revision, not a one-time sign-off. (TU Delft)
2. The crumple zone can move upstream. If accountability lands on an individual architect, we have rebuilt the scapegoat one level up. Green's answer, institutional rather than individual oversight, applies here too.
3. The evidence base is lopsided. The review layer has decades of study: what people catch, what they miss, how quickly they drift toward approving. Evidence that specifying things up front matters exists in safety engineering, in data-centric machine learning, and in small experiments with formal domain schemas, but those fields rarely cite each other. I could not find a study that tests whether designing a system's categories and quality standards up front changes how well oversight works. This article is my reasoned position, and testing it is the obvious next step. If you know of work I have missed, I would like to hear about it.

What to ask instead

Stop asking whether humans are in the loop. Start asking whether humans designed the loop, and what it is standing on.

When someone tells you their system has a human in the loop, four questions separate a safeguard from a comfort:

• What does that human see, and how much time do they have per decision?
• Could they tell if the system were wrong in a way that looked plausible, and how do you know?
• Who defined what counts as quality and evidence, where is it written down, and who can change it?
• What happens to this role as the system improves?

If the answers are "a fraction," "seconds," "probably not," and "it shrinks," then the loop is mostly the machine's, and the foundation under it matters more than the person beside it.

This is the problem I work on.

Jennifer Kinne works on AI governance and epistemic integrity. She is Head of Epistemic Integrity at the Institutional Coherence Initiative, on the Advisory Board of ITAG, and founder of VeracIQ; she has also been in Harvard's FAS for two decades.