SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation

Haomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang University of Notre Dame {hzhuang2, xzhang33}@nd.edu · 2026

Read SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation by Haomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang University of Notre Dame {hzhuang2, xzhang33}@nd.edu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01988.txt.

Description

Zhuang and colleagues test whether language models show number sense with SenseMath, a benchmark of 4,800 items across eight types. It runs three checks: using shortcuts, judging fit or mislead, and generating fresh problems admitting a shortcut. Across five models, prompted models used shortcuts and gained up to 15 percent accuracy on shortcut items. Yet under chain-of-thought prompting spontaneous use ran below 40 percent of cases. Competence stopped at use: models overapplied shortcuts where they fail and could not generate valid ones, showing procedure without understanding of when shortcuts work.

Catalog record

CREATORHaomin Zhuang, Xiangqi Wang, Yili Shen, Ying Cheng, Xiangliang Zhang University of Notre Dame {hzhuang2, xzhang33}@nd.edu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51865
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01988

Browse the full library · About the Tower of Babel