CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
Daniel Kaiser Arnoldo Frigessi Ali Ramezani-Kebrya Benjamin Ricaud · 2025
Read CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density by Daniel Kaiser Arnoldo Frigessi Ali Ramezani-Kebrya Benjamin Ricaud (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18458.txt.
Introduction
Long reasoning tests for language models often mix up three things: how hard the task is, how long it runs, and how much clutter it holds. Daniel Kaiser, Arnoldo Frigessi, Ali Ramezani-Kebrya, and Benjamin Ricaud built CogniLoad to pull those apart. Their 2025 benchmark makes logic puzzles in plain language. Each factor can be tuned on its own, guided by cognitive load theory.
One dial sets the built-in difficulty of the puzzle. A second sets the ratio of distractors to real clues. A third sets task length. The team ran 22 top reasoning models through the full grid. Task length proved the dominant limit on scores. Tolerance for difficulty varied a lot by model. Responses to clutter followed a U shape for many models. The tool gives builders a clean way to find why a model fails.
Catalog record
| CREATOR | Daniel Kaiser Arnoldo Frigessi Ali Ramezani-Kebrya Benjamin Ricaud |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-18458
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density Daniel Kaiser Arnoldo Frigessi Ali Ramezani-Kebrya Benjamin Ricaud Abstract Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad , a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT’s core dimensions: intrinsic difficulty ( d d ) controls intrinsic load; distractor-to-signal ratio ( ρ \rho ) regulates extraneous load; and task length ( N N ) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances to intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogniLoad provides a reproducible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development. 1 Introduction Cognitive Load Theory (CLT) ( Sweller 1988 ) characterizes three types of cognitive load on human working memory when solving problems ( Sweller 1988 ; Paas et al. 2003 ; Lieder and Griffiths 2020 ) : intrinsic (ICL), extraneous (ECL), and germane (GCL). ICL stems from the inherent complexity and element interactivity of the task ( Halford et al. 1998 ) . ECL is induced by suboptimal task presentation requiring the processing of elements that are not task-relevant ( Chandler and Sweller 1991 ) . GCL concerns effective remaining resources allocated to engaging with the intrinsic task demands for mental schema construction ( Ericsson and Kintsch 1995 ; Sweller 2010 ) . Large language models (LLMs) demand analogous computational resources when solving reasoning tasks. The essential element interactivity of a reasoning chain mirrors ICL; distractor elements reflect ECL; and sustained engagement with intrinsically relevant information over a long reasoning process acts as an operational proxy for germane-like processing - the constructive effort to maintain a coherent problem representation. To the best of our knowledge, no study has based the evaluation of problem-solving capacities of LLMs in CLT by distinguishing these three load types, and existing benchmarks often confound them: LongBench ( Bai et al. 2024a ) and L-Eval ( An et al. 2024 ) vary context length but not necessarily the intrinsic reasoning depth; LogicBench ( Parmar et al. 2024 ) probes ICL with minimal demands on ECL or context-induced load; BABILong ( Kuratov et al. 2024 ) mixes multi-step reasoning with fixed distractor ratios, obscuring precise failure attribution. We introduce CogniLoad , a controllable synthetic benchmark for long-context reasoning, inspired by CLT, that operationalizes these load types through tunable parameters in randomized natural-language logic puzzles: (i) Intrinsic Load via Intrinsic Difficulty d controls the number of interacting entities, attributes, and logical clauses, directly manipulating ICL by varying essential element interactivity and reasoning depth. (ii) Extraneous Load via Distractor Density \rho l dictates distractor density; lower \rho increases irrelevant elements, manipulating ECL. (iii) Germane Load Proxy via Task Length N serves as an operational proxy for demanding germane-like cognitive work. Our key contributions are summarized as follows: 1. We ground the evaluation of LLMs in CLT, precisely defining benchmark parameters that control ICL, ECL, and an operational proxy for the conditions conducive to GCL. 2. We introduce CogniLoad , the first benchmark designed to independently control these three dimensions of cognitive load, while scaling to arbitrarily long contexts. 3. We provide an algorithm for the automatic randomized generation and evaluation of puzzle instances, enabling large-scale and reproducible comparison of LLM capabilities. 4. We report empirical results on 22 state-of-the-art (SotA) reasoning LLMs (see Figure 1 ), revealing distinct failure regimes across the (d,N,\rho) dimensions and highlighting specific targets for improving LLM design. Together, these contributions translate CLT into a precise diagnostic framework for understanding and advancing long-context reasoning in LLMs. Figure 1: The average accuracy of models across the evaluated parameter space for d\in\{1,3,5,7,10\} (left panel), N\in\{20,50,100,250\} (center panel), and \rho\in\{5,...,95\} (right panel). Each plot selects one dimension for the X-axis and averages the accuracy of all evaluated puzzles for the other two dimensions relative to it. 1.1 Related work Long-context Benchmarks (Working Memory Capacity). A line of work starting with Long-Range Arena (LRA) ( Tay et al. 2020 ) and followed by several recent benchmarks probe LLM performance on long sequences, often framed as testing “memory load” or context utilization. Earlier studies such as SCROLLS ( Shaham et al. 2022 ) , BookSum ( Kryściński et al. 2021 ) , and QMSum ( Zhong et al. 2021 ) scale document length without manipulating intrinsic difficulty. LongBench ( Bai et al. 2024a ; Bai et al. 2024b ) and L-Eval ( An et al. 2024 ) aggregate multi-task corpora up to 200k tokens, while BABILong ( Kuratov et al. 2024 ) , LongReason ( Ling et al. 2025 ) , RULER ( Hsieh et al. 2024 ) , ZeroSCROLLS ( Shaham et al. 2023 ) , and Michelangelo ( Vodrahalli et al. 2024 ) increase context while the inherent difficulty of individual sub-tasks (ICL) may vary unsystematically and distractor density (ECL) is often not a controlled variable. Consequently, performance degradation could be due to sheer length overwhelming processing capacity, or an inability to sustain germane-like cognitive work over extended relevant information, but the precise cause of failure is not clear. Logical-reasoning Benchmarks (Intrinsic Load). A complementary line of benchmarks focuses on ICL by presenting tasks with high inherent complexity but often within minimal context lengths or distractors. Notable classical suites include ReClor ( Yu et al. 2020 ) , LogiQA ( Liu et al. 2020 ) , and BIG-Bench-Hard (BBH) ( Suzgun et al. 2022 ) . AutoLogic ( Zhu et al. 2025 ) is a benchmark that explicitly focuses on scaling ICL through controllable complexity. LogicBench ( Parmar et al. 2024 ) , CLUTRR ( Sinha et al. 2019 ) , and ZebraLogic ( Lin et al. 2025 ) also exemplify this by formulating symbolic logic puzzles that demand processing many interacting elements (e.g., multi-step deductions, handling negation, and constraint satisfaction). Similarly, mathematical reasoning datasets, e.g., GSM8K ( Cobbe et al. 2021 ) and abstract rule induction tasks, e.g., ARC-AGI ( Chollet et al. 2024 ) primarily escalate ICL by increasing the complexity of essential rules and their interdependencies. Needle in a Haystack Benchmarks (Extraneous Load). Needle in a haystack (NIAH) designs ( Gkamradt 2023 ) specifically target ECL by embedding relevant facts (“needles”) within large volumes of distractor text (“hay”). Variants such as Sequential NIAH ( Yu et al. 2025 ) and Nolima ( Modarressi et al. 2025 ) investigate the impact of such distractors, which constitute non-essential elements requiring processing for filtering, thereby imposing ECL. While these benchmarks effectively isolate the impact of distractors on information retrieval, the “needle” tasks themselves typically involve low ICL (e.g., simple fact lookup). Need for Multi-dimensional Evaluation. CLT highlights the interplay of ICL, ECL, and germane processing under finite working memory ( Paas et al. 2003 ) . Existing LLM reasoning benchmarks, however, typically manipulate only one dimension without systematic, independent control over the