LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

Sumeet Ramesh Motwani, Daniel Nichols, Charles London, Peggy Li, Fabio Pizzati, Acer Blake, Hasan Hammoud, Tavish McDonald, Akshat Naik, Alesia Ivanova, Vignesh Baskaran, Ivan Laptev, Ruben Glatt, Tal Ben-Nun, Philip Torr, Natasha Jaques, Ameya Prabhu, Brian Bartoldson, Bhavya Kailkhura, Christian Schroeder de Witt · 2026

Read LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning by Sumeet Ramesh Motwani, Daniel Nichols, Charles London, Peggy Li, Fabio Pizzati, Acer Blake, Hasan Hammoud, Tavish McDonald, Akshat Naik, Alesia Ivanova, Vignesh Baskaran, Ivan Laptev, Ruben Glatt, Tal Ben-Nun, Philip Torr, Natasha Jaques, Ameya Prabhu, Brian Bartoldson, Bhavya Kailkhura, Christian Schroeder de Witt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14140.txt.

Introduction

Motwani and nineteen colleagues introduce LongCoT, a benchmark of 2,500 expert-built problems across chemistry, mathematics, computer science, chess, and logic. Each problem gives a short input with a checkable answer, but solving it means threading linked steps that can run to hundreds of thousands of reasoning tokens. Since each local step stays within reach of top models, failures expose long-range reasoning limits rather than single-step skill. At release the best models score below 10 percent, leaving a wide gap in current ability.

Catalog record

CREATORSumeet Ramesh Motwani, Daniel Nichols, Charles London, Peggy Li, Fabio Pizzati, Acer Blake, Hasan Hammoud, Tavish McDonald, Akshat Naik, Alesia Ivanova, Vignesh Baskaran, Ivan Laptev, Ruben Glatt, Tal Ben-Nun, Philip Torr, Natasha Jaques, Ameya Prabhu, Brian Bartoldson, Bhavya Kailkhura, Christian Schroeder de Witt
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE114233
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning Sumeet Ramesh Motwani, Daniel Nichols, Charles London, Peggy Li, Fabio Pizzati, Acer Blake, Hasan Hammoud, Tavish McDonald, Akshat Naik, Alesia Ivanova, Vignesh Baskaran, Ivan Laptev, Ruben Glatt, Tal Ben-Nun, Philip Torr, Natasha Jaques, Ameya Prabhu, Brian Bartoldson, Bhavya Kailkhura, Christian Schroeder de Witt Abstract As language models are increasingly deployed for complex autonomous tasks, their ability to reason accurately over longer horizons becomes critical. An essential component of this ability is planning and managing a long, complex chain-of-thought (CoT). We introduce LongCoT, a scalable benchmark of 2,500 expert-designed problems spanning chemistry, mathematics, computer science, chess, and logic to isolate and directly measure the long-horizon CoT reasoning capabilities of frontier models. Problems consist of a short input with a verifiable answer; solving them requires navigating a graph of interdependent steps that span tens to hundreds of thousands of reasoning tokens. Each local step is individually tractable for frontier models, so failures reflect long-horizon reasoning limitations. At release, the best models achieve < < 10% accuracy (GPT 5.2: 9.8%; Gemini 3 Pro: 6.1%) on LongCoT, revealing a substantial gap in current capabilities. Overall, LongCoT provides a rigorous measure of long-horizon reasoning, tracking the ability of frontier models to reason reliably over extended periods. Figure 1 : Accuracy versus token usage on LongCoT. GPT 5.2 achieves 9.83% with an average of 62K output tokens per problem. † † affiliationnotice: * Joint-first authors. † Equal advising. Correspondence: [email protected] , [email protected] [Website] [Benchmark] [Code] Simplified Long-Horizon Reasoning Problems (3 of 5 LongCoT Domains) Computational Structure Chemistry Reaction Cascade 9 subproblems with explicit ( \to ) dependencies # Each step produces/selects a molecule; later reactions combine earlier outputs. N_{1} : Identify molecule from structural properties \to Mol-1. N_{2} : Match structure string to known molecule \to Mol-2. N_{3} : Predict product of Mol-1 + Mol-2 \to Mol-3. combines N_{1},N_{2} [ …identify Mol-4 via formula; select Mol-6, Mol-7 from candidates… ] N_{5} : Predict product of Mol-3 + Mol-4 \to Mol-5. combines N_{3},N_{4} N_{8} : Predict product of Mol-6 + Mol-7 \to Mol-8. combines N_{6},N_{7} N_{9} : Compute molecular diameters for Mol-5, Mol-7, Mol-8. combines N_{5},N_{7},N_{8} Answer: Diameters =[11,5,5] 1 2 3 4 6 5 7 8 9 Mathematics Chained Competition Problem 14 subproblems with explicit ( \to ) dependencies # 14 competition problems chained: each answer parameterizes the next. N_{1} : Consecutive integers with product divisible by k . backtrack: k depends on N_{2} and N_{11\text{--}14} , creating a cyclic dependency. N_{2} : Nested-squares geometry; branches into two chains. uses N_{1} N_{3} : Find a volume \to N_{4} : Evaluate a sum. from N_{2} [ … N_{5}\to N_{6}\to N_{7} : further competition problems… ] N_{8} : Find roots \to N_{9} : Count divisors \to N_{10} : Count reflections. N_{9}\!\leftarrow\!N_{4} ; N_{10}\!\leftarrow\!N_{8} N_{11} : Optimize \to N_{12} : Primes \to N_{13} : Angles \to N_{14} . from N_{2} ; N_{14}\!\leftarrow\!N_{12} Answer: (N_{10},N_{4},N_{1},N_{13})=(9,5,5,27) 1 2 3 11 4 5 6 12 7 8 13 14 9 10 backtrack Chess Minimax Pawn Capture Problem has implicit subproblems with dependencies # The dependency graph is a game tree that emerges from the rules. Setup: 30\times 30 board with a knight at (7,13) and 8 pawns at given positions. Game: Two players alternate turns choosing which pawn the knight captures next (moving via shortest path). Alice must maximize total moves; Bob minimizes. Find the total moves under optimal play. Search: 8! orderings naively; prunable via minimax with memoization. Answer: 106 moves \BlackKnightOnWhite p_{1} p_{2} p_{7} p_{8} \cdots p_{1} p_{2} p_{7} p_{8} \cdots p_{1} p_{3} p_{4} p_{5} p_{6} p_{7} p_{8} p_{1} p_{3} p_{4} p_{6} p_{7} p_{8} \vdots depth 8 Figure 2 : LongCoT problems demand long-horizon reasoning. Each of our five domains requires constructing and traversing a computational dependency graph in a long chain-of-thought. These graphs can be DAGs, search trees, cyclic graphs, constraint graphs, or execution traces. Frontier models struggle with LongCoT problems: even the best model (GPT 5.2) achieves only 9.83% accuracy. Table 1 : Comprehensive benchmark comparison. LongCoT uniquely combines short inputs ( < 6K tokens), long reasoning outputs (10K–100K+ tokens), controlled single-step difficulty, no tool dependencies, and verifiable answers across five diverse domains. Existing benchmarks either test short reasoning chains, require long inputs, depend on tools, or do not control for step-level difficulty. Benchmark Input Long-Horizon Scalable Reasoning Task Best Context Reasoning Needed Difficulty Isolation Domain Score Frontier Reasoning FrontierMath ( 10 ) Short ( \leq 10K tok) ✗ ✓ (HC) Math 31.3% HLE ( 21 ) Short ( \leq 8K tok) ✗ ✓ (HC) Multi 38.3% FrontierScience ( 30 ) Short Unknown ✗ ✓ (HC) Multi 25.2% Long-Context Benchmarks LongBench v2 ( 4 ) Long ✗ ✓ ✗ Multi > 65% MRCR v2 ( 29 ) Long ✗ ✓ ✗ Synthetic > 95% Agentic Benchmarks SWEBench-Verified ( 12 ) Short ✓ ✗ ✗ (SC) Code 80.9% WebArena ( 36 ) Short ✗ ✗ (SC) Functional 71.6% BrowseComp ( 31 ) Short ✗ ✗ (SC) Web 77.9% Gaia2 ( 8 ) Short ✓ ✗ ✗ (SC) Multi 42.1% PaperBench ( 25 ) Long ✓ ✗ ✗ (HC) AI Papers > 40% TerminalBench ( 19 ) Short ✓ ✗ ✗ (HC&SC) Code 64.9% HCAST ( 22 ) Short ✓ ✗ ✗ (SC) Code N/A LongCoT (Ours) Short ✓ ✓ ✓ Multi 9.83% hi Reasoning Isolation: HC = hardness confounded as success also requires esoteric knowledge; SC = scaffold confounded as success also requires tool use. 1 Introduction The ability of language models to reason reliably over long chains of thought becomes critical as they are deployed on increasingly harder tasks. Current benchmarks measure this capability only indirectly, either through hard but short reasoning problems or through agentic workflows where tool use and scaffolds leverage the underlying models’ reasoning abilities. As context limits grow and test-time scaling is adopted widely, an important frontier emerges: the fundamental ability of models to maintain a coherent reasoning trace over long horizons. In this work, we design a benchmark to measure and study this capability in order to ask: How accurately can frontier models reason over long horizons in their chain of thought? We define long-horizon reasoning as the ability to reason reliably over many interdependent steps in an extended chain of thought. Complex tasks require models to (i) plan, explore, and backtrack; (ii) maintain long-term state by prioritizing relevant context; (iii) monitor progress and discover errors or dead-ends; and (iv) perform credit assignment by linking errors or progress to specific prior steps. Current benchmarks rarely stress-test these abilities. FrontierMath ( 10 ) caps generations at 10K tokens. HLE ( 21 ) typically induces fewer than 5K reasoning tokens on average. Agentic benchmarks evaluate complex multi-step workflows, but domain-specific tool use and scaffolding dominate improvements ( 19 ) . Long-context benchmarks ( 4 ; 17 ) test retrieval over long inputs but require only short outputs. What remains an open problem is directly measuring the inherent capabilities of models to keep reasoning as their chain of thought grows longer. To this end, we introduce LongCoT, a benchmark of 2,500 problems from chemistry, mathematics, computer science, chess, and logic. Each problem consists of a short prompt (median 2K tokens) with a verifiable final answer; solving it requires navigating a graph

Read the complete text