Truth as a Compression Artifact in Language Model Training
Konstantin Krestnikov · 2026
Read Truth as a Compression Artifact in Language Model Training by Konstantin Krestnikov (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11749.txt.
Introduction
Krestnikov asks why language models trained on clashing data still lean toward correct answers. His tests use small transformers, 3.5M to 86M parameters, on math problems shown with right and wrong solutions. When errors are random, accuracy climbs from 65 to 85 percent with model size. When errors follow one coherent false rule system, accuracy falls to chance. A second false rule, oddly, restores most of the truth bias. He proposes a Compression-Consistency Principle: training favors the most compressible answer cluster, not truth itself.
Catalog record
| CREATOR | Konstantin Krestnikov |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46068 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11749
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Truth as a Compression Artifact in Language Model Training Konstantin Krestnikov Abstract When transformers train on contradictory data—the same problem with both correct and incorrect solutions—which answer do they prefer? We hypothesize that next-token prediction, as a compression process, favors whichever answer cluster has lower description length; truth benefits only when errors lack internal structure. We test this by training transformers (3.5M–1B parameters) from scratch on controlled corpora, systematically varying the structure of errors. We find that (a) when errors are random, models develop a correctness preference scaling from 65% to 85% with model size; (b) when errors follow a single coherent alternative rule, this preference vanishes ( ∼ 45 {\sim}45 – 51 % 51\% ); (c) two competing wrong rules suffice to restore it ( 47 % → 78 % 47\%\to 78\% ). The pattern reproduces on Wikipedia paragraphs with entity substitution (71% vs 46%) and at 1B scale on a mixed natural-text corpus (77% vs 47%). These results are consistent with the hypothesis that, in controlled contradictory corpora, model preference tracks the relative compressibility of competing answer systems rather than truth per se. 1 Introduction Real-world training corpora are noisy: the same question may receive contradictory answers across different documents. Yet language models trained on such data tend to prefer correct information. If we do not understand why this filtering works, we cannot predict when it will fail—for instance, when errors are not diverse but internally consistent. Several explanations have been offered: scaling improves factual performance ( Kadavath et al., 2022 ) ; frequency matters, since factual accuracy correlates with how often correct information appears in training data ( Elazar et al., 2022 ; Joshi et al., 2024 ; Kandpal et al., 2023 ) ; and truth-correlated internal representations have been found ( Burns et al., 2023 ; Marks and Tegmark, 2024 ; Bürger et al., 2024 ) . Kalai and Vempala (2024) established that calibrated models must hallucinate at a rate bounded by the corpus’s monofact rate. Yet none of these explain why the training objective alone would favor one answer over another when both appear at equal frequency in the same format. We propose that the answer lies in the structure of errors. Minimizing cross-entropy is equivalent to minimizing code length ( Shannon, 1948 ; Deletang et al., 2024 ) , linking LLM training to the Minimum Description Length principle ( Rissanen, 1978 ; Grünwald, 2007 ) . In our task families, a correct rule system is compact; diverse random errors must be memorized individually. But a coherent false system—internally consistent, just wrong—can be equally compact, and the preference vanishes. We test this hypothesis in a controlled experimental setting. We train transformers (3.5M–1B parameters) from scratch on corpora where each mathematical problem appears with both correct and incorrect solutions, systematically varying the structure of errors. This design isolates error structure as the independent variable while holding frequency, format, and domain constant. Our contribution is a single experimentally supported hypothesis: in controlled contradictory corpora, the compression objective favors consistency, not truth. We show that (a) random errors produce a correctness preference scaling from 65% to 85%; (b) a single coherent alternative rule eliminates this preference; (c) two competing rules restore it, pinpointing the compressibility boundary; (d) the same contrast reproduces on Wikipedia paragraphs with entity substitution; and (e) the effect persists at 1B parameters on a mixed natural-text corpus (Section 4.7 ). Whether this extends to large-scale pretraining remains open (Section 5 ). 2 Related Work Prediction as compression. The link between prediction and compression traces to Shannon (1948) and was formalized by Solomonoff (1964) and Rissanen’s MDL principle ( Rissanen, 1978 ; Grünwald, 2007 ) . Deletang et al. (2024) showed that LLMs are universal compressors. Huang et al. (2024) discovered a linear correlation ( r\approx-0.93 ) between compression quality and benchmark performance. Wan and Mei (2025) proved that LLM training approximates Solomonoff induction. Pan et al. (2025) linked compression to knowledge acquisition and scaling; Chlon et al. (2025) linked compression failures to hallucination. Contemporaneous work by Marty et al. (2026) formalizes simplicity bias through MDL, and Aksenov et al. (2026) connects compression to mathematical modeling. We build on the MDL framework by experimentally varying the description length of false answer systems. Internal representations and world models. Compression can give rise to structured internal representations: Li et al. (2023a) found board representations in an Othello model, Gurnee and Tegmark (2024) discovered linear space-time representations in Llama-2, and several studies found truth-correlated representations ( Marks and Tegmark, 2024 ; Burns et al., 2023 ; Li et al., 2023b ; Azaria and Mitchell, 2023 ; Ravfogel et al., 2025 ) . Our work operates at the behavioral level: we identify data-level conditions under which compression produces a preference for correct completions, leaving activation-level analysis for future work. Truthfulness and data statistics. Joshi et al. (2024) linked truthfulness to “personas” in pretraining data. Elazar et al. (2022) demonstrated frequency dependence. Kandpal et al. (2023) showed a direct relationship between document count and accuracy. Li et al. (2024) studied LLM preferences under conflicting knowledge and found that formality and surface quality predict which version the model favors—a consistency-driven explanation complementary to ours. A growing body of work studies knowledge conflicts directly: Xie et al. (2024) surveyed approaches to resolving conflicts between parametric and contextual knowledge, and Longpre et al. (2021) showed that entity-substituted evidence can override parametric memory. Unlike these analyses, we fix frequency, format, and surface quality, varying only the compressibility of the error system itself—isolating error structure as the causal variable rather than studying post-hoc conflict resolution. Simplicity bias and noisy labels. Neural networks prefer simple functions ( Valle-Perez et al., 2019 ; Mingard et al., 2021 ; Goldblum et al., 2024 ) . The noisy labels literature directly parallels our setup: Zhang et al. (2017) showed memorization of random labels; Rolnick et al. (2017) showed robustness to massive label noise. Grokking connects to compression as a memorization-to-generalization transition ( Nanda et al., 2023 ; Liu et al., 2023 ) . Our experiments extend this line by showing that “structured noise”—coherent errors—is not filtered out. To our knowledge, systematic variation of error compressibility in a denoising setting has not been studied before. 3 Methodology Overview. We test the hypothesis that next-token prediction favors whichever answer system has lower description length, regardless of its truth value. To isolate this, we train models from scratch on controlled corpora where each problem appears with contradictory answers, varying only the structure of errors. The core experiments compare random errors (high description length) against coherent errors (low description length) at equal frequency (Section 4.1 ), then probe the boundary with multi-rule errors of intermediate compressibility (Section 4.2 ). Transfer to natural language is tested via Wikipedia entity substitution (Section 4.3 ). 3.1 Hypothesis and MDL Framing Consider a corpus where each problem appears with a correct answer (theory T_{1} ) and an alternative ( T_{2} ). An MDL learner ( Rissanen, 1978 ; Grünwald, 2007 ) minimizes L(M)+L(D|M) . When K(T_{2})\gg K(T_{1}) (random errors), the false system’s description length grows with corpus size, favoring T_{1} . When K(T_{2})\approx K(T_{1}) (coherent errors), both are compact and the learner has no basis to prefer one