Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information

Youcheng Huang, Bowen Qin, Chen Huang, Duanyu Feng, Xi Yang, Wenqiang Lei, Jiancheng Lv · 2025

Read Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information by Youcheng Huang, Bowen Qin, Chen Huang, Duanyu Feng, Xi Yang, Wenqiang Lei, Jiancheng Lv (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11252.txt.

Description

Youcheng Huang and co-authors test reasoning models on problems that lack key facts in a 2025 paper. Large reasoning models score well on math tests, but those tests always give complete problems. A real helper should notice when facts are missing and ask for them, rather than guessing ahead. That skill is rarely measured.

The team built a data set of incomplete problems in varied settings. Across models, the systems rarely asked for the missing facts. Instead they often over-thought the problem or made up facts that were never given. Extra training taught the skill in part, but with clear limits. The study argues that asking good questions should count as a core part of machine intelligence.

Catalog record

CREATORYoucheng Huang, Bowen Qin, Chen Huang, Duanyu Feng, Xi Yang, Wenqiang Lei, Jiancheng Lv
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11252

Browse the full library · About the Tower of Babel