SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation
Haomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang University of Notre Dame {hzhuang2, xzhang33}@nd.edu · 2026
Read SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation by Haomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang University of Notre Dame {hzhuang2, xzhang33}@nd.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01988.txt.
Description
Zhuang and colleagues test whether language models show number sense with SenseMath, a benchmark of 4,800 items across eight types. It runs three checks: using shortcuts, judging fit or mislead, and generating fresh problems admitting a shortcut. Across five models, prompted models used shortcuts and gained up to 15 percent accuracy on shortcut items. Yet under chain-of-thought prompting spontaneous use ran below 40 percent of cases. Competence stopped at use: models overapplied shortcuts where they fail and could not generate valid ones, showing procedure without understanding of when shortcuts work.
Catalog record
| CREATOR | Haomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang University of Notre Dame {hzhuang2, xzhang33}@nd.edu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51865 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01988