When LLMs Stop Following Steps: A Diagnostic Study of Arithmetic Procedural Execution in Language Models

Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh · 2026

Read When LLMs Stop Following Steps: A Diagnostic Study of Arithmetic Procedural Execution in Language Models by Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00817.txt.

Description

Sailesh Panda, Pritam Kadasi, Abhishek Upperwal, and Mayank Singh test whether language models truly follow step-by-step arithmetic procedures, rather than just reaching plausible answers. Their diagnostic benchmark gives a model a defined procedure and two numbers, and asks for the final value. Difficulty grows through longer procedures and through steps that must look back at earlier intermediate results.

Average first-answer accuracy fell from well over half on five-step procedures to about one fifth on ninety-five-step procedures. Looking at full outputs, failures often meant missing answers, answers given too early, self-correction after an initial error, or traces that stopped short. The pattern shows steady decline as procedural complexity rises.

Catalog record

CREATORSailesh Panda, Pritam Kadasi, Abhishek Upperwal, Mayank Singh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00817

Browse the full library · About the Tower of Babel