Trapped in the past? Disentangling fluid and crystallized intelligence of large language models using chess
Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsäcker · 2026
Read Trapped in the past? Disentangling fluid and crystallized intelligence of large language models using chess by Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsäcker (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16823.txt.
Introduction
Leonard Pleiss, Maximilian Schiffer, and Robert von Weizsäcker use chess to test what chatbots truly know. Chatbots ace many tests, but why is unclear. They may recall vast text by heart. Or they may reason from first rules each time. Their 2026 preprint sorts these two gifts with chess boards.
Chess gives a clean test bed with strong judge tools. The team sorts boards from famed to brand new. Famed boards can be solved by recall of past games. New boards call for fresh thought on the spot. Gaps in scores then show where recall ends. Such tests help labs build bots that truly think.
Catalog record
| CREATOR | Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsäcker |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16823
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Trapped in the past? Disentangling fluid and crystallized intelligence of large language models using chess Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsäcker Abstract Large Language Models (LLMs) exhibit remarkable capabilities, yet it remains unclear to what extent these reflect sophisticated recall (crystallized intelligence) or reasoning ability (fluid intelligence). We introduce chess as a controlled testbed for disentangling these faculties. Leveraging the game’s structure and scalable engine evaluations, we construct a taxonomy of positions varying in training corpus proximity–ranging from common states solvable by memorization to novel ones requiring first-principles reasoning. We systematically evaluate multiple GPT generations under varying reasoning intensities. Our analysis reveals a clear gradient: performance consistently degrades as fluid intelligence demands increase. Notably, in out-of-distribution tasks, performance collapses to random levels. While newer models improve, progress slows significantly for tasks outside the training distribution. Furthermore, while reasoning-augmented inference improves performance, its marginal benefit per token decreases with distributional proximity. These results suggest current architectures remain limited in systematic generalization, highlighting the need for mechanisms beyond scale to achieve robust fluid intelligence. † † footnotetext: Preprint. K eywords Large Language Models, Fluid Intelligence, Generalization, Out-of-distribution performance 1 Introduction Large Language Models (LLMs) have achieved remarkable progress in recent years, setting new state-of-the-art results across a broad range of benchmarks and applications ( 17 ) . This rapid advancement has led to widespread adoption in both research and industry ( 21 ; 18 ) . Yet, the origins of these performance gains are still under debate. It remains unclear whether they stem from improvements in genuine problem-solving and reasoning capability–commonly attributed to fluid intelligence ( 3 ) –or from enhanced memorization and retrieval across ever-expanding training corpora, commonly attributed to crystallized intelligence ( 3 ) . This distinction is crucial: while both can yield impressive scores on specific benchmarks, only fluid intelligence enables an agent to generalize to novel problems and systematically reason beyond its prior experience ( 5 ) . Understanding whether LLMs primarily rely on crystallized or fluid intelligence therefore carries direct implications for the trajectory, safety, and interpretability of their reasoning and behavior. However, empirically disentangling these two forms of intelligence remains a major challenge for two reasons. First, measuring intelligence itself is difficult, as no universally accepted, unitary definition exists. To circumvent this theoretical difficulty, we adopt the common, yet simplifying assumption that intelligence is process-invariant and can therefore be assessed through task performance ( 5 ) . We explicitly acknowledge that this operational definition narrows the scope of inquiry to measurable functional competence rather than cognitive equivalence and refer to Appendix F for a detailed discussion on our framework’s validity. Second, independently assessing fluid and crystallized intelligence poses an additional challenge: It requires evaluating models on tasks that differ in their proximity to the training data distribution. Performance on within-distribution (WD) tasks, i.e., tasks that were contained in the training data, is indicative of crystallized intelligence, whereas out-of-distribution (OOD) performance is indicative of fluid intelligence. Yet, because modern LLMs are trained on massive and largely opaque datasets, quantifying this distributional proximity typically remains out of reach. As a result, existing studies have relied on isolated benchmarks that often conflate retrieval, pattern matching, and reasoning, yielding fragmented evidence, leading to an ongoing debate on the nature of intelligence in LLMs (see Section 2 ). We address this gap by proposing a controlled empirical framework that leverages chess as a structured environment for studying strategic reasoning in LLMs . As a domain combining clear rules, deep combinatorial structure, and computationally verifiable performance metrics via chess engines, chess provides a rigorous testbed for disentangling fluid and crystallized intelligence: All chess games originate from a single initial state, but the number of reachable configurations grows combinatorically–approximately 10^{120} possible games exist ( 25 ) –ensuring that most games and positions are effectively novel after only a few moves. Therefore, chess also engages both forms of intelligence: early-game play benefits from crystallized intelligence, i.e., recalling known openings, whereas succeeding in later game stages often demands fluid intelligence, i.e, finding novel solutions in previously unseen situations (for details, see Appendix C ). This dual nature–extremely common positions in early stages of the game, yet complete novelty in later stages–enables systematic variation in the likelihood that a given position was encountered during training. Extensive open-source game databases permit frequency-based estimation of positional familiarity, providing partial control over data contamination ( 16 ; 14 ; 15 ) . Finally, state-of-the-art chess engines offer scalable move evaluations ( 20 ; 27 ; 22 ) . While not being absolute ground-truths, these engines serve as high-resolution baselines for computational correctness. We thus utilize them to quantify decision regret relative to a calculated and approximated optimum (see Appendix F ). In this paper, we conduct a systematic evaluation of LLM performance in the structured domain of chess, across tasks that require differing degrees of fluid and crystallized intelligence. Although our empirical analysis focuses on chess, its implications extend to broader settings of strategic reasoning in formal systems, as the game embodies general principles of structured problem-solving such as abstraction, planning, and adaptation within explicit constraints. Our intent is not to advocate for or against LLM capabilities, but to provide rigorous, domain-grounded evidence that can inform ongoing discourse on the intellectual nature of these systems and their developmental trajectory. Our results reveal a consistent performance gradient: accuracy declines as the need for OOD generalization increases. In positions that are unlikely to have been observed during training, model performance collapses toward random play. Given the impossibility of strictly precluding data leakage, these results suggest that our estimates likely represent an upper bound on true fluid capability. While newer generations of models exhibit steady improvement, gains diminish as tasks leave the proximity of the training data distribution. Reasoning-augmented modes provide measurable benefits, but the marginal benefits per token decrease with decreasing training distribution proximity. Collectively, these findings highlight persistent limitations in fluid intelligence, suggesting that current LLM architectures may face inherent challenges in meaningfully generalizing beyond memorized knowledge. 2 Related work With this investigation, we aim to contribute to an active debate on the extent to which LLMs exhibit emergent reasoning and generalization–phenomena closely related to the notion of fluid intelligence. The distinction between crystallized and fluid intelligence parallels long-standing research on WD versus OOD performance and on generalization in neural networks. In machine learning terms, crystallized intelligence corresponds to performance grounded in recall within the support of the training distribution, while fluid intelligence reflects compositional generalization under limited priors ( 5 ) . An increasing number of studies explore the degree to which current LLMs can generalize beyond their training data distribution. However, the evidence remains inconclusive. Some results point to genuine generalization–or even emergent capabilities ( 30 ; 8 ; 28 ; 1 ; 26 ; 2 ; 10 ; 9 ) . For instance, 26 found that LLM -generated research ideas were rated as more novel than those of human experts, while 10 observed that LLMs succeed in spatial reasoning tasks with negligible data overlap, suggesting the presence of genuine generalization. Conversely, other work highlights persistent limitations in reasoning or generalization ( 5 ; 4 ; 31 ; 6 ) . The Abstraction and Reasoning