21 Oct 2026 6 min read Practical

Poisoning the Well Is an Integrity Problem

The worry

At a recent conference, several business owners described the same fear: an employee puts something into the company's AI system, and false conclusions come back out. The security literature calls this data poisoning and treats it as an attack. That framing fits part of the problem and misses most of it.

An employee does not have to be malicious. A wrong number, an outdated policy, or an unverified file dropped into a shared folder does the same damage, because the system cannot tell a mistake from sabotage. This piece makes four points. The three ways bad input arrives are one failure. What the research shows about poisoned models is narrower than the headlines suggest. Provenance and corroboration answer different questions. And corroboration is much weaker inside a company than it looks.

Three doors, one failure

Bad input reaches an AI system through three doors, and the output looks the same from each.

Malicious. Someone deliberately plants false material: a doctored record, a fabricated document, a corrupted training example.
Accidental. Someone in good faith enters a wrong figure, uploads a draft, or pastes in something unverified. This is far more common than sabotage.
Stale. The input was true when it went in and has since stopped being true. A superseded policy sits in the index next to its replacement.

From the system's side these are indistinguishable. A fluent, confident answer comes back, and nothing in it signals which door the content came through.

Security researchers study the first door. Knowledge-management vendors write about the third, usually as content rot. The two literatures rarely meet, even though one attack technique, CorruptRAG, works by labeling the true answer as outdated and presenting the attacker's answer as current. That is exactly what a stale document does by accident.

Treating the three as one problem changes the question. It stops being "how do we stop attackers?" and becomes "how does this system decide what to believe, and who can change that?"

A trigger is not a belief

The best-known result in this area is a study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute. It found that 250 poisoned documents were enough to plant a backdoor in models ranging from 600 million to 13 billion parameters, and that model size and the volume of clean training data made no difference. The headlines read: any AI model can be poisoned with a few hundred files.

The backdoor in the study was simple. When a rare trigger phrase appeared in a prompt, the model produced gibberish. The authors describe the experiment as designed around low-stakes behavior, and the models tested were small by current standards.

That distinction matters. A trigger is a switch: a pattern the model learns to pair with a behavior, dormant until the pattern appears. A belief is different. A model that has seen the sky described as blue across millions of contexts will not come to think it is purple every other Friday because a few hundred documents say so. The study did not test anything like that, and the weight of well-attested evidence makes a strong prior a much harder target than a gibberish switch.

But that protection only covers facts the model has overwhelming evidence for. It does nothing for the facts a business depends on most: its refund policy, a supplier's terms, an internal threshold, the current version of a protocol. For those, one document may be all the evidence the system has.

Retrieval removes the protection

Most businesses are not training models. They connect a model to their documents, a design called retrieval-augmented generation, or RAG. The model searches the document store, reads what comes back, and answers from it.

Retrieval removes the dilution that protects a trained model. There is no crowd of good patterns to outvote a bad one, because the model is reading, not recalling. If a bad document is the closest match to a question, it is the evidence. A USENIX Security 2025 paper, PoisonedRAG, showed that injecting five crafted texts per target question into a knowledge base of millions of texts produced the attacker's chosen answer about 90% of the time. Later work, such as AuthChain, got there with a single document written in authoritative language.

Honest errors work the same way. One analysis of enterprise deployments describes a case where retrieval did its job, found the most relevant document, and the answer was wrong anyway, because better retrieval surfaces stale documents as readily as current ones.

In a retrieval system, the "exact document asked for" case is not a rare exception. It is the normal path.

Provenance and corroboration answer different questions

Two defenses come up in nearly every discussion, and they are often treated as interchangeable. They are not.

Provenance: where did this come from? It records who added a piece of content, when, and how. It does not say whether the content is true, and a well-meaning employee's wrong document has perfect provenance. What provenance buys is remediation. When something turns out to be bad, you can find everything that source touched and roll it back, and you can trace an answer to its origin.

Corroboration: is this claim supported by anything else? It treats agreement among sources as evidence. Research systems such as MIRAGE score a claim by how many diverse sources support it and how consistent they are, and fall back on the model's own knowledge when the sources look unreliable. Unlike provenance, it speaks to truth.

Neither does the other's job. Provenance without corroboration lets you trace a falsehood perfectly. Corroboration without provenance lets you act on agreement you cannot audit. A system needs both, and needs to know which question it is asking.

Corroboration needs independence, and companies rarely have it

Corroboration is evidence only when the sources are independent. Two witnesses who heard the story from the same person are one witness.

Outside a company this is already a known weakness. One engineering write-up on agent memory notes that "two independent sources" is the retrieval version of an old, unsolved problem: if an attacker can create sources cheaply, with new domains, new document IDs, and fresh timestamps, they can manufacture their own corroboration.

Inside a company the problem is different, and arguably worse, because nobody has to manufacture anything. Internal documents descend from one another. The policy lives in the handbook, then in a slide deck copied from the handbook, an FAQ written from the deck, a wiki page summarizing the FAQ, and a training document built from the wiki. If the original is wrong or out of date, all five agree, and a corroboration check counts five votes for the error. The same happens with templates, forwarded emails, and any AI-generated summary that gets written back into the knowledge base. The agreement comes from shared ancestry, not shared evidence.

That is why raw agreement counts are the wrong measure inside an organization. What matters is how many independent lines of origin support a claim. Measuring that takes lineage: knowing which documents were derived from which, and collapsing a family of copies into a single vote. It also takes sources that sit outside the copy chain altogether, such as the system of record, the signed contract, the measurement, or the person who owns the fact.

Provenance, then, is more than an audit trail. It is the input that makes corroboration possible. Without lineage, a system cannot tell consensus from echo.

What to do about it

None of this needs new technology. It needs the discipline companies already apply to financial records, applied to what an AI system is allowed to believe.

Treat knowledge-base changes like code changes. Attribution, review, and approval for anything high-stakes, with different people introducing and approving a change.
Record lineage, not just authorship. Mark every copy, summary, and derivative as derived, and keep a pointer to what it came from.
Count independent origins, not matching documents. Collapse a family of copies into one vote, and weight systems of record above prose that describes them.
Give facts an owner and an expiry date. Stale is a governance state, not a search problem. Someone should be accountable for each fact that drives a decision.
Keep the system from writing its own corroboration. Label AI-generated summaries and exclude them from evidence counts.
Surface conflict instead of resolving it silently. When sources disagree, show the disagreement. Humans filter stale documents well when they can see two versions side by side. A single packaged answer hides that.
Test with planted errors. Insert a controlled false document in a test environment and watch whether, where, and how it surfaces.
Keep a person in the loop for consequential decisions, and spot-audit outputs against ground truth.

The common thread is that a company's AI system is only as trustworthy as its answer to one question: why does it believe this? If the answer is "it was in the folder," the system has no defense against any of the three doors.