Large Language Models and Mathematical Reasoning Failures
Johan Boye, Birger Moëll · 2025
Read Large Language Models and Mathematical Reasoning Failures by Johan Boye, Birger Moëll (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11574.txt.
Description
Johan Boye and Birger Moell graded this 2025 test of language models on fifty fresh high-school math word problems. Prior tests mostly checked final answers. This team read the working too, scoring the reasoning steps of eight leading models, including recent reasoning-tuned ones. Newer models answered more problems correctly, yet every model tripped somewhere.
Errors clustered in spatial reasoning, planning, and plain arithmetic. Some right answers arrived through wrong logic. Models made unwarranted assumptions, chased surface number patterns, and struggled to turn physical intuition into equations. Multi-step and real-world problems hurt most, despite the models' broad knowledge. The authors caution that answer accuracy alone overstates machine problem-solving, and urge grading the reasoning itself.
Catalog record
| CREATOR | Johan Boye, Birger Moëll |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11574