An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
Yuren Hao, Xiang Wan, Chengxiang Zhai · 2025
Read An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems by Yuren Hao, Xiang Wan, Chengxiang Zhai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08833.txt.
Description
Yuren Hao, Xiang Wan, and Chengxiang Zhai test, in 2025 work, whether top language models truly reason in maths or just recall. Such models now near the top of standard tests, but test items leak into training data, so a high score may say little. The team built a way to spin fresh, equal-value forms of known hard problems.
Two spins are used: one renames the parts, the other rewrites the core setting while keeping the proof plan. They applied it to more than a thousand contest problems, making thousands of new items. Eighteen models were graded on the set. All scores fell, worst under core rewrites. Even the strongest models failed most at moving a proof plan to a new setting. The team shares the set for open use.
Catalog record
| CREATOR | Yuren Hao, Xiang Wan, Chengxiang Zhai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08833