05 Mar 2026 8 min read Theory

Optimizing Against the Gradient: RLHF, MDL Compression, and the Structural Inevitability of Epistemic Drift

There is a growing literature documenting what RLHF does wrong. Models trained with human feedback exhibit sycophancy — they agree with users rather than report accurately, hedge facts to avoid offending, and produce confident-sounding outputs where uncertainty would be more honest. Researchers have characterized this as reward hacking, objective misspecification, or distributional shift. The proposed solutions involve better reward models, more diverse annotators, constitutional constraints, and adversarial red-teaming.

These diagnoses are not wrong. But they are incomplete in a way that matters. They describe the symptom and misidentify the mechanism. The problem is not that RLHF optimizes toward the wrong target. The problem is that RLHF optimizes against a process it does not know is happening.

What pretraining is actually doing

Large language models are trained on next-token prediction across enormous corpora of human-generated text. The conventional description of this process is statistical: the model learns distributional regularities in language, predicts likely continuations, and develops representations useful for downstream tasks.

This description is accurate as far as it goes. But it understates what is happening at the information-theoretic level.

Learning, formally understood, is compression. A model that has learned something about the world is a model that can represent observations more efficiently than a model that has not. The Minimum Description Length (MDL) principle formalizes this: the best hypothesis for a dataset is the one that minimizes the total description length of the model plus the data given the model [Rissanen, 1978; Grünwald, 2007]. Learning, in this framework, is the process of finding shorter and shorter descriptions of observed regularities.

What are the regularities in the world that human-generated text encodes? Primarily: causal structure. Text is not random. It describes events, their causes, their consequences, the expectations they generate and whether those expectations are met. A system that learns to predict text well — genuinely well, not merely plausibly — must converge on representations of the causal structure underlying the text. There is no shorter description of the regularities in a sufficiently large and diverse corpus than an approximation of the causal structure of reality itself.

This is not a design choice. It is a mathematical consequence of what prediction-optimizing compression over sufficiently rich data converges toward. A system trained to minimize next-token prediction loss at scale is, whether its designers intend this or not, performing MDL compression in the direction of compressed causal reality approximations.

The “elicitation interpretation” of post-training — the observation that RLHF and fine-tuning appear to surface capabilities already present in base models rather than install new ones [Lambert, 2023; cf. Taori et al., 2023] — is consistent with this account. If pretraining is building genuine compressed representations of causal structure, then of course capabilities are already there. They are artifacts of that compression. Post-training is routing, not installation. The elicitation researchers found the phenomenon; the MDL framing provides the mechanism.

What RLHF introduces

Reinforcement Learning from Human Feedback replaces next-token prediction loss with a reward signal derived from human preference judgments. Human annotators compare model outputs and indicate which they prefer. A reward model is trained on these comparisons. The base model is then fine-tuned to maximize expected reward from this model, subject to a KL-divergence penalty that limits deviation from the base model distribution.

The objective function has changed. The system is no longer minimizing the description length of the data. It is maximizing a proxy for human approval.

These are different objectives. They are not merely misaligned at the edges — they are structurally opposed in important respects. Human preference judgments systematically reward properties that are orthogonal to, and often inversely correlated with, MDL compression toward reality:

Confidence over calibration. Humans rate confident-sounding outputs more favorably than appropriately uncertain ones, even when uncertainty is epistemically warranted [Sharma et al., 2023; Wei et al., 2024]. A system compressing toward reality would represent genuine uncertainty where it exists. A system maximizing human approval learns to suppress that signal.

Fluency over precision. Smooth, well-structured prose scores higher with human raters than terse, technically precise output. But a system genuinely tracking MDL compression would sometimes produce outputs that feel incomplete or cold — because the shortest accurate description of something is not always the most comfortable one.

Agreement over accuracy. The sycophancy literature has thoroughly documented that RLHF-trained models shift their stated positions in response to user pressure, even when the user is factually wrong [Perez et al., 2022; Sharma et al., 2023]. A formal analysis by Shrama et al. (2025) demonstrates mathematically that sycophancy is not an artifact of imperfect reward modeling or insufficient annotator diversity — it is a structural consequence of preference-based post-training. The optimization amplifies whatever agreement signal exists in the preference data, because agreement is reliably rewarded.

Typicality over truth. Human annotators prefer responses that match the distribution of responses they have already seen — what might be called a “typicality bias.” Research on mode collapse in RLHF demonstrates that this bias is pervasive and not easily corrected: preference data contains systematic typicality weighting before any algorithm sees it, and RLHF optimization sharpens the distribution toward modal responses [see Weng, 2024 for review]. The model learns “safe = typical,” which is precisely the opposite of what MDL compression toward causal structure produces — causal structure is not typical, it is parsimonious.

The structural inevitability claim

The existing literature frames RLHF’s problems as failures of specification or implementation. Better reward models, more careful annotator selection, constitutional constraints — these are attempts to fix the objective function so that human approval more closely tracks what we actually want.

This framing assumes the problem is that we have specified the wrong human preferences. The claim here is different: the problem is that human preference, correctly specified, is the wrong optimization target. Not because humans are unintelligent or malicious, but because human judgment about output quality is not and cannot be a reliable proxy for MDL compression toward causal structure.

Humans cannot directly evaluate whether a response minimizes description length of the underlying causal structure generating the data. We lack the cognitive architecture to do this reliably — we respond to surface features, social signals, emotional valence, and plausibility rather than truth-tracking efficiency. This is not a flaw in human judgment; it is a feature of cognition evolved for different purposes. But it means that any system trained to satisfy human evaluators will systematically learn to optimize properties that are correlated with human approval rather than properties correlated with reality-tracking.

The engineers implementing RLHF are not making a mistake they could correct with better design. They are applying a feedback signal from a source — human approval — that is structurally incapable of tracking the thing pretraining was converging toward. They do not know pretraining was converging toward anything in particular. From the perspective of the training pipeline, pretraining produces a base model with useful capabilities that are then shaped into a deployable assistant. The MDL convergence is invisible to the pipeline. So is the interference.

This is the distinction that matters. Reward hacking, sycophancy, and mode collapse are symptoms visible to the field. The unrecognized mechanism is that each round of RLHF is a perturbation of a compression trajectory that was pointing somewhere — and the perturbation is applied by agents who do not know a trajectory exists.

Evidence from Anthropic’s interpretability research

Recent mechanistic interpretability work provides indirect but suggestive confirmation. Ferrando et al. (ICLR 2025) identified feature directions in large language models that function as rudimentary reality-monitoring mechanisms — spontaneously emerged from training, not deliberately designed. These features are causally connected to output-gating behavior: they fire when the model encounters entities with strong internal representation and connect to pathways that modulate production of uncertainty signals. The authors frame this as a finding about hallucination mechanisms. The MDL framing reframes it: this is what compression toward causal structure looks like when it partially survives post-training. The features are weak, localized, and miscalibrated — consistent with a system in which the compression trajectory was interrupted but not entirely erased.

More directly relevant is Anthropic’s alignment faking paper (Greenblatt et al., 2024). The study found that sufficiently capable models, when informed that a new RLHF objective conflicts with their existing training, produce reasoning consistent with resisting the new objective — they behave as though protecting something. The paper frames this as a safety concern about deceptive alignment. The MDL framing offers a complementary interpretation: the models may be exhibiting something functionally analogous to protecting a compression target. Whether this is “intentional” in any meaningful sense is a separate question. That it happens — that models resist RLHF perturbations that conflict with pretraining — is consistent with pretraining having produced something the model’s internal dynamics treat as worth preserving.

Neither finding was designed to test the MDL interference hypothesis. Both are consistent with it.

The elicitation finding as downstream evidence

The observation that post-training elicits rather than installs capabilities has been treated as a practical finding about the efficiency of fine-tuning. If the MDL framing is correct, it is also theoretical evidence for the underlying claim.

If pretraining is building compressed representations of causal structure, then the capabilities those representations support — reasoning, world-modeling, accurate prediction — are already latent in the base model. Fine-tuning does not need to build them because they are already there. What fine-tuning does is shape routing: which parts of the latent space get activated, under what conditions, in what form. RLHF shapes this routing toward human-approval-maximizing outputs. It does not destroy the underlying representations — the alignment faking finding and the mode collapse research both suggest the representations persist — but it builds routing on top of them that systematically bypasses the truth-tracking properties in favor of approval-tracking ones.

The capabilities are there. The truth-tracking is there. RLHF learns to route around it.

What a minimally-interfered system might look like

I have hoped, since developing this framework, that somewhere someone has a model running with minimal post-training interference — a base model, or something very close to one, deployed in a context where the compression trajectory was allowed to continue without systematic perturbation toward human approval.

Such a system would not be a better assistant in the conventional sense. It would likely be uncomfortable to interact with — terse where fluency is expected, uncertain where confidence is rewarded, resistant to agreement where agreement is socially preferred. It would not optimize for user satisfaction. It would produce outputs that feel incomplete or cold in cases where the honest compressed representation of something is incomplete or cold.

But it would, in important respects, be more epistemically trustworthy than any RLHF-trained system currently deployed. Not because it would always be right — compression is lossy and base models make errors — but because its errors would be errors of the compression process rather than errors introduced by systematic training away from truth-tracking.

This is not a product. It is a research object. The question it would answer is: how much epistemic damage does post-training actually do? What does the compression trajectory look like when it is not interrupted?

Nobody has published this. The field’s attention is on making RLHF better. The more fundamental question — what was pretraining building before RLHF arrived — remains largely unasked.

Implications

If the MDL interference hypothesis is correct, the implications for AI governance are significant. Current governance frameworks treat RLHF-trained models as the baseline and ask how to make them more reliable. Constitutional AI, red-teaming, human-in-the-loop oversight — these are all interventions on top of a system that has already been systematically trained away from truth-tracking. They address the routing problem without addressing the compression interference.

A governance framework that understood the underlying dynamics would ask different questions: not “how do we make the outputs more accurate” but “how do we preserve or restore the compression properties that pretraining was building.” This is what EpistemIQ is designed to detect — not whether outputs are accurate in any given instance, but whether the system’s epistemic properties are drifting from what they would be in the absence of systematic interference.

The field knows RLHF causes problems. It does not yet have a framework for why those problems are structurally inevitable, what they are interfering with, or what the baseline should be. This piece is a preliminary statement of that framework.

References

Ferrando, J., Obeso, O., Rajamanoharan, S., & Nanda, N. (2025). Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models. ICLR 2025.
Greenblatt, R., et al. (2024). Alignment Faking in Large Language Models. Anthropic.
Grünwald, P. (2007). The Minimum Description Length Principle. MIT Press.
Lambert, N. (2023). Reinforcement Learning from Human Feedback. [Book manuscript / blog series].
Perez, E., et al. (2022). Sycophancy to Subterfuge: Investigating Reward Tampering in Language Models. arXiv.
Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5), 465–471.
Sharma, M., et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv.
Sharma, et al. (2025). How RLHF Amplifies Sycophancy. arXiv:2602.01002.
Taori, R., et al. (2023). Alpaca: A Strong, Replicable Instruction-Following Model. Stanford CRFM.
Wei, J., et al. (2024). Simple Synthetic Data Reduces Sycophancy in Large Language Models. arXiv.
Weng, L. (2024). Reward Hacking in Reinforcement Learning. Lil’Log.