Context Is Not Comprehension
Alex Pan, Mary-Anne Williams · 2025
Read Context Is Not Comprehension by Alex Pan, Mary-Anne Williams (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04907.txt.
Introduction
Alex Pan and Mary-Anne Williams argue in a 2025 paper that long input does not equal real understanding in large language models. Top models can recall facts from very long texts almost perfectly. Yet recall can hide weak step by step reasoning about states that never appear in plain words. The authors build a new test that hides list based math steps inside story text.
Their test checks every middle step, so it shows where a model's chain first goes wrong. Models that score near perfect on the plain task fail fast on the hidden version, within ten thousand tokens. The test builder can hide other checkable tasks too, such as logic or symbol work. Readers learn a sharper way to judge true grasp.
Catalog record
| CREATOR | Alex Pan, Mary-Anne Williams |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04907
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Context Is Not Comprehension Alex Pan, Mary-Anne Williams Abstract The dominant way of judging Large Language Models (LLMs) has been to ask how well they can recall explicit facts from very long inputs. While today’s best models achieve near-perfect recall, this masks a harder skill: performing multi-step reasoning and tracking intermediate state that never appears verbatim. We introduce Verbose ListOps (VLO), a benchmark that embeds deterministic ListOps computations inside narrative camouflage and, crucially, allows step-level evaluation of every intermediate result. Experiments show that models which solve raw ListOps with ≈ 100 % \approx 100\% accuracy collapse on VLO after only 10 k tokens. By exposing where a model’s reasoning chain first diverges, VLO moves assessment beyond sheer context length and toward genuine comprehension. VLO’s generation pipeline is task-agnostic: it can weave any deterministically verifiable reasoning schema—arithmetic, symbolic, abductive, inductive or defeasible—into narrative form. This makes VLO a reusable test-bed for the next wave of reasoning-centric model designs, not merely those with step-explicit scaffolds. 1 Introduction Despite million-token context windows, today’s LLMs still stumble on tasks that a human finds trivial: following a multi-step argument buried inside a distracting story ( Liu et al. 2024 ; Shi et al. 2023 ; An et al. 2023 ) . This ability—to ignore irrelevant detail, maintain intermediate conclusions, and synthesise them later—is the backbone of knowledge work, from a lawyer interpreting clauses to a sales manager inferring intent from chatty transcripts. Yet benchmarks rarely measure it. To bridge this gap, we introduce Verbose ListOps (VLO), a novel benchmark that isolates this ability in a controlled setting, challenging models to reason, not just locate information. Existing long-context suites chiefly test (i) factual recall in "needle-in-a-haystack" settings ( Kamradt 2023 ; Li et al. 2024b ) or (ii) shallow reading comprehension ( An et al. 2023 ; Zhang et al. 2024 ) . They do not control reasoning depth, nor do they hide intermediate answers, so models can succeed through pattern matching rather than computation. Verbose ListOps (VLO) fills this gap. ListOps is a synthetic benchmark where models perform nested arithmetic operations on sequences of integers following a tree structure—for example, [MAX 3 7 [MIN 1 9 5] 2] requires computing the inner MIN first, then finding the maximum among all values. We convert each ListOps node into a short story fragment, hide every intermediate value behind a ’narrative anchor’, and add semantically related distractors. The resulting benchmark forces a model to compute, cache, and reuse internal state. Figure 1: VLO embeds ListOps problems in a narrative, forcing LLMs to find and solve each subproblem, track intermediate states, and ignore distractors for the final answer, emulating complex human text synthesis. Takeaway: VLO reveals LLMs needs more than just long-context processing—they must compute within distracting narratives where intermediate results remain implicit. In this paper, we deliver three key contributions: 1. A broadly-applicable, agentic generation framework. We open-source a "World-Author–Critic–Validator" pipeline that can embed any deterministically-verifiable reasoning graph—not only arithmetic ListOps but also symbolic, abductive, inductive, or defeasible logic (see Appendix A ). 2. Verbose ListOps (VLO): a first instantiation of the framework. Using this pipeline we build VLO, a benchmark that narrativises nested ListOps computations. VLO is the first long-context dataset that (i) withholds all intermediate results, (ii) controls context length and reasoning depth orthogonally, and (iii) ships exact intermediate-step labels, enabling systematic failure-mode analysis. 3. Orthogonal control knobs for context and reasoning complexity. Tree depth, operator arity, narrative length, and distraction density are exposed as independent hyper-parameters, allowing researchers to stress-test scaling behaviour along a single axis while holding all others fixed—something prior benchmark suites cannot do. 4. A diagnostic evaluation toolkit and empirical study. We release scripts that check correctness at every anchor step, plot error-propagation curves, and compare prompting regimes. Applying this toolkit to ten prominent LLMs uncovers a consistent blind spot: models that excel at raw ListOps or factual recall often diverge at the very first hidden step when the same logic is wrapped in narrative camouflage. Together, the framework plus toolkit turn VLO from a one-off dataset into a reusable test-bed for any future long-context reasoning architecture. Our experiments show state-of-the-art LLMs, despite solving ListOps with ease, on VLO, suffer a severe ( \approx 50%+) drop in accuracy at just \approx 10k-token contexts . The result exposes an under-tested limitation: LLMs struggle to synthesize multi-step conclusions amongst narrative distraction. VLO offers a rigorous, open-source tool for diagnosing (and improving) this foundational capability, seeking to move discussion beyond simply enlarging context windows. 2 Related Work The expansion of LLM context windows has catalyzed research into their performance on extensive textual inputs, leading to a diverse landscape of evaluation benchmarks. Benchmark Primary Reasoning Task Nested Reasoning Chains Distraction Tunable Reasoning Difficulty Scalable Context Deterministic Answer Realistic Tasks Generation Method Verbose ListOps (Ours) Algorithmic (ListOps) embedded in narrative ✓ Coherent, semantically relevant narrative ✓ ✓ Mathematically deterministic ✓ Agentic LongReason ( Ling et al. 2025 ) General QA (Comprehension, Inference, Maths) \times Irrelevant passages embedded around relevant context \approx ✓ Multiple-choice (via reasoning) ✓ Context expansion Needle-in-Haystack Type Factual Recall \times Vast irrelevant corpus \times ✓ Exact match \times Target insertion Other Synthetic (e.g. RULER Hsieh et al. 2024 .) Code/Instruction Following \times Structured, less narrative \times ✓ Task-specific \times Template Based Human Annotated Long-Context QA General QA \times Natural document structure \times \times Human-judged ✓ Human annotation Table 1: Comparison of Verbose ListOps with other long-context benchmarks. Using an agentic generation process, Verbose ListOps offers both controllable context lengths and reasoning difficulty. 2.1 Long-Context Benchmarks Early long-context evaluations typically adapted standard NLP benchmarks ( Shaham et al. 2023 ; An et al. 2023 ; Bai et al. 2024 ) . These early benchmarks often had contexts shorter than the maximum capabilities of current LLMs ( Gemini Team 2024 ; OpenAI 2025a ; Anthropic 2025 ) and inadequately differentiated complex reasoning tasks from simpler tasks based on factual recall tasks or failed to analyze the impact of distractors sufficiently ( Muhlgay et al. 2023 ) . Recent synthetic benchmarks have addressed some of these limitations by allowing greater control over context length. The "needle-in-a-haystack" (NIAH) paradigm, exemplified by NeedleBench ( Li et al. 2024b ) , specifically evaluates an LLM’s recall capability within extensive irrelevant contexts, focusing on fact extraction rather than complex multi-step reasoning. Other synthetic benchmarks, such as RULER ( Hsieh et al. 2024 ) , test specific reasoning capabilities like variable tracking and multi-hop information extraction. However, their artificial scenarios may not adequately replicate the challenges posed by naturalistic narratives with coherent distractors ( Haller et al. 2024 ) . Benchmarks such as InfiniteBench ( Zhang et al. 2024 ) or those utilizing extensive document curation like LooGLE ( Li et al. 2024a ) often rely heavily on human annotation or semi-automated methods. While realistic, these approaches are labor-intensive and limit scalability and precise control over task variables such as context length and complexity, underscoring the need for standardized, synthetic evaluation frameworks ( Valmeekam et al. 2023 ; Muhlgay et al. 2023 ) . LongReason ( Ling et al. 2025 ) significantly advances synthetic benchmarks by offering variety—reading comprehension, logical inference, and mathematical reasoning tasks are expanded into longer, distractor-rich texts. It