Local Fluency, Global Incoherence
"Coherence" is having a moment in AI governance. It shows up in framework names, tool names, pitch decks, LinkedIn bios. Everyone's system is coherent. Everyone's methodology is coherent. Almost none of these claims come with a definition, and the few that do usually mean something weaker than the word implies.
This is more than a semantic quibble. If "coherence" can mean anything from "the slides flow logically" to "the model's outputs don't contradict each other under adversarial pressure," then the word has stopped doing any evaluative work. It's become a stamp of approval you attach after the fact, not a property you test for beforehand. That's a problem in any field, but it's a particular problem in AI governance, because "coherent" is increasingly used as a synonym for "trustworthy" or "safe".
So here is what coherence actually means, mathematically; where the popular usage diverges from it; and where the rigorous version still isn't enough on its own.
The actual definition
Take a set of claims, components, or outputs, and call them X₁ through Xₙ. Their coherence, in the information-theoretic sense, is:
C(X₁,...,Xₙ) = ΣH(Xᵢ) − H(X₁,...,Xₙ)
In words: add up how much it costs to describe each part on its own (its individual entropy, H(Xᵢ)) and then subtract how much it actually costs to describe all of them together. This is total correlation, or multi-information, and it has an exact operational meaning: it's the number of bits you save by using one code built for the joint distribution instead of separate optimal codes for each part. If the parts are statistically independent, joint description costs exactly the sum of the parts: no savings, C = 0. If they're mutually load-bearing; that is, if knowing some of them lets you compress your description of the rest, that gap is real and positive. This isn't a metaphor for compression. It is compression, in the exact Shannon sense: bits saved by joint coding over independent coding.
This is worth sitting with because it reframes what a coherence claim actually commits you to. To say a framework, a dataset, or a set of institutional claims is coherent is to say: if I already know some of these, I need meaningfully less information to specify the rest. That's a falsifiable, computable claim. Most things that get called coherent were never checked against it.
A worked example
Suppose three binary claims, A, B, and C, each either true (1) or false (0), each individually a coin flip: P(true) = 0.5.
Case 1: genuine dependency. Suppose B is logically entailed by A (wherever A holds, B holds too) and C is unrelated. Each variable on its own still carries 1 bit of entropy: H(A) = H(B) = H(C) = 1, so the sum is 3 bits. But because B just mirrors A, the joint distribution over (A, B, C) only has four possible states instead of eight, (0,0,0), (0,0,1), (1,1,0), (1,1,1), each equally likely. Four equally likely states cost log₂(4) = 2 bits to specify jointly. Total correlation: 3 − 2 = 1 bit. That bit is a quantifiable measure of how much A and B are actually locked to each other. C contributes nothing, because it's independent.
Case 2: no dependency. A, B, C are independent coin flips, but presented as three interlocking pillars of one framework. ΣH is still 3 bits. With no dependency, all eight joint states are equally likely, so H(A,B,C) = 3 bits. Total correlation: 3 − 3 = 0 bits. Zero, no matter how confidently the document asserts these three things reinforce each other.
Two things can be taken from this, one obvious and one not. The obvious one: fluency and coherence answer different questions, and only one is checkable by ear, and that's the point the rest of the piece is about. The less obvious one is this: I chose the joint distribution in both cases. Case 1 is coherent because I, the author, decided B = A. Nothing about the math forced that; I stipulated it. Which raises an uncomfortable question the toy example can't answer on its own.
The problem with a closed set
Here's what actually bothers me about stopping at total correlation: a well-constructed fabrication can score exactly as high as a true account.
A good conspiracy theory is highly coherent in the total-correlation sense: every claim supports several others, remove one and the structure visibly weakens; the swap test described below would pass it with room to spare. So would a well-built novel. Neither is true. And the reason the math can't catch this isn't a flaw in the formula, rather, it's what the formula is built to detect. Total correlation measures statistical dependency among the parts of a set. It says nothing about where that dependency came from. If you are the sole author of every claim in the set, you are also, in effect, the entire generative process the formula is trying to characterize. You can engineer C(X) to be as high as you like, the same way a novelist engineers a plot where every early detail pays off later. That's not cheating the math. It's exactly what authorship over a closed set lets you do.
This is the same objection philosophy has leveled at coherentism as a theory of truth for a long time. Internal consistency alone can't distinguish a well-built fiction from an accurate account of the world; here it is reconstructed in bits instead of in argument. Total correlation doesn't resolve that old problem. It just makes it precise enough to see clearly, which is progress, but not the same as solving it.
The extension test
The fix isn't a better formula for the closed set. It's refusing to evaluate coherence as a property of a closed set at all.
A true account and a well-built fabrication can look identical on the claims their author chose to include. They diverge on what happens when the boundary widens to claims the author didn't get to pick: held-out facts, adversarial questions, data pulled from somewhere else entirely. A structure that's coherent because it tracks something real stays cheap to extend, because the same underlying process that generated the original claims also generated the new ones: the dependency was discovered, not authored, so it keeps paying off outside the original set. A structure that's coherent because an author tuned it to be stays expensive to extend: new claims don't fit cleanly, and maintaining the appearance of coherence requires a growing pile of exceptions and patches. That's structurally the same failure mode as an epicycle-laden model. It is a beautiful fit to the data it was built on, and increasingly baroque the moment you ask it to explain data it wasn't built on. It's also, more plainly, overfitting: a fit with enough free parameters (and an author writing every claim in a closed set has about as many free parameters as exist) will always look good in-sample. Only performance out-of-sample, on claims the author didn't control, tells you whether the dependency was real.
There's a small practical tell hiding in this, worth keeping separately from the formal test: a closed, invented structure is often suspiciously clean. No loose ends, no residue the account doesn't fully absorb, because the author smoothed them away. Real generative structure usually leaves some noise behind that a tidier, authored version wouldn't. Total correlation that looks too clean on a bounded, author-controlled set is a reason for more suspicion, not less.
This is also why I built VeracIQ the way I did rather than as an internal-consistency checker. A system that only evaluates whether its own outputs are consistent with each other is evaluating a closed set it fully controls: precisely the condition under which coherence is cheapest to fake. What's needed instead is an anchor exogenous to the system being evaluated: something the system can't retune to match its own claims, the way physics doesn't retune itself to match a wrong theory just because the theory is well argued. The compression-theoretic measures I use aren't there to score how well a system's claims fit each other. They're there as one unified way of asking the extension question: does this keep compressing when it's tested against something the system didn't get to author.
Local predictability vs. global compressibility
There's a second gap, likely more familiar to the reader, that is separate from the closed-set problem above.
Almost everything currently called "coherent" is coherent in the sense of local predictability: low conditional entropy from one element to the next. A fluent LLM output has this property at the level of tokens in that each word is well-predicted by the words just before it. A polished framework or pitch deck has it at the level of sentences and slides; each claim seems to follow naturally from the last. That smoothness is real, and it's exactly what we're detecting, pattern-matching on, when something "reads as coherent." Human readers are very good at tracking local transitions and very bad at holding a forty-claim structure in working memory well enough to check whether claim four and claim forty actually constrain each other.
But local predictability is a claim about neighbors. Global compressibility (the ΣH(Xᵢ) − H(X₁,...,Xₙ) quantity) is a claim about the whole structure at once. These are different mathematical objects, and one does not imply the other. You can construct (generative models do construct) long passages that are locally smooth from beginning to end while being globally close to zero-coherence in the total-correlation sense.
This is, in fact, close to how large language models produce what's usually called fluent hallucination, though I think "hallucination" is the wrong word for this, for closely related reasons I've written about at more length in Open-Loop Generation. Each token is a strong conditional prediction given recent context, which is exactly what makes it read as coherent, while the document as a whole can contain claims in section one and section six that share almost no real dependency. They could be independently swapped for other claims without the rest of the structure noticing. Fluency was never coherence's proxy to begin with; it just looks like one to a reader moving forward one sentence at a time.
The same mechanism operates in institutional and consulting contexts with no AI involved. A framework document can be internally smooth while none of its components are actually load-bearing for the others. You could remove the third pillar entirely and the first and fifth would be unaffected. That's the tell. Real coherence produces load-bearing structure; local fluency produces the appearance of load-bearing structure, which is cheaper to manufacture and much harder for a reader to distinguish from the real thing without doing the check.
Where existing attempts get partway there
This is not a new problem, and a couple of existing efforts are worth naming, because they get at something real without quite landing on the same target.
Integrated Information Theory's Φ (phi) attempts to measure something structurally adjacent: how much a system's causal structure is more than the sum of its parts, in the specific service of quantifying consciousness. It's a more sophisticated measure than total correlation in some respects, because it tries to capture causal integration rather than mere statistical dependency. But it's built for a different question (is this system an integrated whole in a way that grounds subjective experience?) and inherits enormous computational and definitional difficulty as a result. So it's not something you can casually apply to a governance framework or a set of institutional claims.
Karl Friston's free energy principle, and the Markov blanket formalism it depends on, gets at a related intuition from another direction: organisms and systems maintaining a statistical boundary that separates internal states from external ones, with internal coherence maintained by minimizing prediction error across that boundary. But the formalism has a circularity worth flagging: specifying a Markov blanket requires a pre-specified causal graph, which means you have to already know the system's causal structure (including, implicitly, something like spacetime and directional causal ordering) before you can use the framework to characterize that structure. It presupposes part of what it's meant to reveal. I'm working on causal ordering elsewhere.
Total correlation doesn't solve every problem those frameworks are reaching for. It's a coarser, more tractable measure. But that coarseness is a feature for this purpose: it's computable, it doesn't require solving consciousness or presupposing a causal graph, and it directly answers the question that actually matters when someone claims a framework, a dataset, or a governance structure is "coherent": does knowing part of it actually reduce your uncertainty about the rest, or does it just read like it should?
A test you can run without the math
You don't need to compute entropies to apply this. Here's the practical version.
Don't read a framework, a deck, or a set of claims start to finish and ask whether it flows because that tests local predictability, and it will almost always say yes, because that's what editing for fluency optimizes. Instead, pull two components from opposite ends of the structure, like the founding premise and a downstream conclusion three sections later, or the first module and the last, and ask a narrower question: does one genuinely constrain the other? If you changed the first, would the second have to change? If you could swap either one out for something else without anything downstream noticing, you don't have coherence. You have sequence.
This is a cheap, repeatable test, and it's uncomfortable to apply to your own work, which is exactly why everyone should apply it. Most frameworks, including good ones, have some components more load-bearing than others. That's fine. What's not fine is a framework where nothing is load-bearing, dressed in transitions smooth enough that nobody checks.
Why the word is worth defending
None of this is an argument against fluency, or against writing well, or against frameworks that read persuasively. It's an argument for keeping two different properties (how something reads and what something is) distinct, especially in a field where "coherent" has started functioning as an unearned proxy for "correct," "safe," or "trustworthy."
A claim about coherence is a claim about structure: that the parts of a thing constrain each other in a way that's checkable, in principle, by asking whether knowing some of it reduces uncertainty about the rest. That's a high bar. It should stay a high bar. The word is doing real work only as long as most things that get called coherent would fail the test if you actually ran it. Unfortunately, right now, most of them would.
Further reading: Open-Loop Generation, on why "hallucination" is the wrong word for the related generation-time failure this piece touches on.
More from Jen
