Large Language Models and Mathematical Reasoning Failures

Johan Boye, Birger Moëll · 2025

Read Large Language Models and Mathematical Reasoning Failures by Johan Boye, Birger Moëll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11574.txt.

Description

Johan Boye and Birger Moell graded this 2025 test of language models on fifty fresh high-school math word problems. Prior tests mostly checked final answers. This team read the working too, scoring the reasoning steps of eight leading models, including recent reasoning-tuned ones. Newer models answered more problems correctly, yet every model tripped somewhere.

Errors clustered in spatial reasoning, planning, and plain arithmetic. Some right answers arrived through wrong logic. Models made unwarranted assumptions, chased surface number patterns, and struggled to turn physical intuition into equations. Multi-step and real-world problems hurt most, despite the models' broad knowledge. The authors caution that answer accuracy alone overstates machine problem-solving, and urge grading the reasoning itself.

Catalog record

CREATORJohan Boye, Birger Moëll
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11574

Browse the full library · About the Tower of Babel