SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models

Sebastiano Monti [email protected] Ipazia SpA, Italy, Carlo Nicolini [email protected] Ipazia SpA, Italy, Giovanni Pellegrini [email protected] Ipazia SpA, Italy, Jacopo Staiano [email protected] University of Trento, Italy, Bruno Lepri [email protected] Fondazione Bruno Kessler, Ipazia · 2026

Read SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models by Sebastiano Monti [email protected] Ipazia SpA, Italy, Carlo Nicolini [email protected] Ipazia SpA, Italy, Giovanni Pellegrini [email protected] Ipazia SpA, Italy, Jacopo Staiano [email protected] University of Trento, Italy, Bruno Lepri [email protected] Fondazione Bruno Kessler, Ipazia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20856.txt.

Introduction

Seljon Akhmedli constructs examples that separate two views of order in dynamical systems. The 2026 paper concerns nilfactors, a key structure in ergodic theory. A system can be described through its measurable nilfactor or its topological nilfactor. The two need not match. Akhmedli asks which combinations of behavior can occur. The answer is all of them, and the paper builds systems to prove it.

The examples are minimal and uniquely ergodic. Minimal means every orbit fills the space. Unique ergodicity means time averages settle to one outcome. The construction adapts an idea from Furstenberg. He built a system on the two-torus that was minimal but not uniquely ergodic. The new work extends that trick to control both nilfactors at once. It completes a picture that theorists use to classify complex motion.

Catalog record

CREATORSebastiano Monti [email protected] Ipazia SpA, Italy, Carlo Nicolini [email protected] Ipazia SpA, Italy, Giovanni Pellegrini [email protected] Ipazia SpA, Italy, Jacopo Staiano [email protected] University of Trento, Italy, Bruno Lepri [email protected] Fondazione Bruno Kessler, Ipazia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20856

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models Sebastiano Monti [email protected] Ipazia SpA, Italy, Carlo Nicolini [email protected] Ipazia SpA, Italy, Giovanni Pellegrini [email protected] Ipazia SpA, Italy, Jacopo Staiano [email protected] University of Trento, Italy, Bruno Lepri [email protected] Fondazione Bruno Kessler, Ipazia SpA, Italy Abstract Although the capabilities of large language models have been increasingly tested on complex reasoning tasks, their long-horizon planning abilities have not yet been extensively investigated. In this work, we provide a systematic assessment of the planning and long-horizon reasoning capabilities of state-of-the-art Large Reasoning Models (LRMs). We propose a novel benchmark based on Sokoban puzzles, intentionally simplified to isolate long-horizon planning from state persistence. Our findings reveal a consistent degradation in planning performance when more than 25 moves are required to reach the solution, suggesting a fundamental constraint on forward planning capacity. We show that equipping LRMs with Planning Domain Definition Language (PDDL) parsing, validation, and solving tools allows for modest improvements, suggesting inherent architectural limitations which might not be overcome by test-time scaling approaches alone. 1 Introduction Automated Planning, i.e. the task of generating sequences of actions to achieve a goal, is a well-studied problem in the field of Artificial Intelligence (AI) ( 10 ) , since it requires AI systems to exhibit cognitive abilities such as reasoning, understanding, and efficient state space search ( 51 ) . To this end, automated planning literature has focused on the use of formal languages, such as the Planning Domain Definition Language (PDDL) ( 26 ; 34 ; 12 ) ), and of tree-search strategies or specific heuristics to find optimal solutions ( 4 ) . Large Language Models (LLMs) and, in particular, Large Reasoning Models (LRMs) i.e., LLMs trained to produce so-called reasoning traces resembling structured thought processes, have demonstrated impressive capabilities in natural language understanding, knowledge retrieval and multi-modal pattern recognition ( 16 ; 11 ; 45 ) . However, recent studies highlighted the limitations of such models when applied to planning tasks ( 49 ; 38 ) . For instance, internal reasoning processes have been shown to resemble a form of wandering through the solution space rather than a systematic exploration ( 24 ) . This distinction becomes particularly important for problems that require maintaining sequential state information, such as spatial exploration in constrained environments. In these settings, effective tracking of working memory is necessary to infer the agent’s latent previous state ( 54 ) . In this work, we investigate the long-horizon planning abilities of LRMs using a highly simplified variant of the Sokoban puzzle ( 7 ) . Rather than increasing spatial complexity, we deliberately minimize the structural complexity of the environment while preserving the long-horizon nature of the task by creating examples with the lowest possible branching factor compatible with solvability: a single movable block placed within a linear corridor with tightly controlled geometry. This setting allows us to isolate long-horizon planning from state persistence: models are required to produce complete solution sequences without external memory, intermediate feedback, or state validation, relying solely on internal state representations to track the evolving environment. We therefore investigate to what extent LRMs can sustain coherent planning over long (but simple) action sequences and whether even minimal reasoning branching in otherwise trivial Sokoban instances is sufficient to induce planning failures. Concretely, we examine whether current LRMs can reliably solve linear-corridor Sokoban puzzles with minimal possible branching and identify the point at which increases in horizon length lead to catastrophic breakdowns in action validity, despite the simplicity of the underlying environment. As we will show these minimal sub-problems which are trivial to humans ( 17 ) , are still challenging for Large Reasoning Models as shown by other preliminary studies involving spatial intelligence ( 6 ) . We posit this as a systemic deficiency in long-term action representation and sequential logic, and in spatial reasoning and thus as an important limitation of current LRMs that is not yet fully understood. 2 Related Work 2.1 Benchmarks for LLM Planning As mentioned above, planning requires LLMs to blend logical, numerical, and spatial reasoning with long-horizon strategic adaptation, rather than just relying on pattern matching or memorization. Classical planning domains expressed in or derived from the Planning Domain Definition Language (PDDL ( 9 ) ), such as BlocksWorld ( 40 ) , Towers of Hanoi and similar tasks ( 33 ) , remain a common benchmark choice, though earlier attempts date back to the pre-ChatGPT era ( 39 ) . Test suites like PlanBench ( 47 ) introduced structured, domain-agnostic evaluations inspired by classical planning ( 10 ) , including plan generation ( 32 ; 50 ; 22 ) and optimality ( 47 ; 53 ; 48 ) . In another line of work, planning is evaluated within agentic or workflow-based frameworks, where LLMs are required to decompose goals into multiple sub-plans ( 27 ; 55 ; 22 ) . The results in these settings are encouraging though highly cost intensive. Importantly, when not equipped with external tools or made part of larger workflows (e.g., enabling stateful tracking ( 15 ) ), innate planning abilities remain still weak ( 35 ) . Even the latest foundational models are found to consistently fail in delivering correct sequences of actions (in any format or language) due to two primary deficits: weak internal state representations leading to invalid moves and misleading heuristic search resulting in loops or early termination, as shown in the textual game “8-puzzle” in 35 . Moreover, efficacy of different prompting techniques is model-dependent in a non-predictable way ( 35 ; 8 ) . Other works have systematically investigated the performances of LLMs in playing textual games with gym -style APIs ( 5 ; 14 ) . Beyond structured puzzles, community-driven and informal game-oriented benchmarks like word-game bench ( 41 ) and nonogram logic puzzles ( 3 ; 21 ) with multi-difficulty instances have been devised to measure how well models plan under both explicit and implicit constraints, track environment states, and adapt over multiple turns. The varying depth of planning ability required helps to reveal how performance scales with complexity and structure. In general, existing benchmarks using specific planning languages and/or internal reasoning traces expressed in natural language show that LLMs exhibit limited planning abilities in various domains ( 19 ) , especially as the complexity and horizon length of the problems increase. This gap motivates the development of new benchmarks tailored to planning and solving structured textual puzzles with LLMs. 2.2 Sokoban as a Benchmark for Planning The Sokoban puzzle involves spatial planning in a highly constrained environment. Solvable Sokoban maps can be generated efficiently ( 29 ) , and the environment is fully controllable and deterministic. These properties enable rigorous evaluation using exact solvers and verifiers, as well as metrics such as search depth and solution time ( 17 ; 37 ) . Unlike puzzles such as the Tower of Hanoi, which can be solved by repeating a simple pattern for larger instances, Sokoban offers no shortcuts. Each map is unique, and moving a single box can block or open paths in ways that prevent a one-size-fits-all solution. As a result, Sokoban is considered a good benchmark for evaluating planning abilities in the 2023 edition of

Read the complete text