How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem

Chun Zheng, Lianlong Wu, Bingqian Li, Lvting Liu, Yi Zhou · 2026

Read How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem by Chun Zheng, Lianlong Wu, Bingqian Li, Lvting Liu, Yi Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06882.txt.

Description

Chun Zheng and four co-authors test language models on the equivalence class problem, possibly the simplest long-chain reasoning task. Given a set of stated equalities between variables, the model must decide whether two variables end up equal, which requires chaining many steps. The team varies variable counts, connection density, and prompts across reasoning and non-reasoning models.

Non-reasoning models fail the task outright. Reasoning models do far better yet still cannot solve it completely. Intriguingly, the hardest instances differ by model type. For plain models they sit at a known phase transition of the random equality graph. For reasoning models they sit at the largest chain diameter. That split separates confusion from genuine reasoning depth.

Catalog record

CREATORChun Zheng, Lianlong Wu, Bingqian Li, Lvting Liu, Yi Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06882

Browse the full library · About the Tower of Babel