Understanding Formal Reasoning Failures in LLMs as Abstract Interpreters
Jacqueline Mitchell, Brian Hyeongseok Kim, Chenyu Zhou, Chao Wang · 2025
Read Understanding Formal Reasoning Failures in LLMs as Abstract Interpreters by Jacqueline Mitchell, Brian Hyeongseok Kim, Chenyu Zhou, Chao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12686.txt.
Description
Jacqueline Mitchell and three co-authors asked how language models reason when they check code, in a paper published in 2025. Models now draft loop proof facts for verification tools, but their thought path was a black box. Two prompt styles were built to draw out a set style of step-by-step thought, the kind that proof craft uses. The team ran top models on 22 small programs from a known verification set.
Each invented proof fact was graded for soundness: would it hold if the code ran? The wrong ones were sorted by theme to find the kinds of slips that recur. The readings show where model reasoning parts from true step logic. The authors aim the findings at two fronts: safer use of models in verification work, and training that fixes the recurring slips.
Catalog record
| CREATOR | Jacqueline Mitchell, Brian Hyeongseok Kim, Chenyu Zhou, Chao Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12686