LLMs Know More About Numbers than They Can Say

Fengting Yuchi Li Du Jason Eisner · 2026

Read LLMs Know More About Numbers than They Can Say by Fengting Yuchi Li Du Jason Eisner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-07812.txt.

Description

Language models stumble when comparing numbers written in different notations, like scientific notation against plain decimals. Fengting Yuchi and two co-authors ask whether the models secretly know the magnitudes even when their spoken answers go wrong.

Probing the hidden states of smaller open models, they find a single linear projection recovers log-magnitudes with about 2.3 percent relative error on synthetic text and 19 percent on scientific papers, and a linear classifier ranks paired numerals with over 90 percent accuracy. Yet the models' explicit answers score only 50 to 70 percent. Adding the probe's loss during fine-tuning lifts verbal accuracy by 3.22 percent, showing internal knowledge can be coaxed into speech.

Catalog record

CREATORFengting Yuchi Li Du Jason Eisner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-07812

Browse the full library · About the Tower of Babel