Evaluating Grounded Reasoning by Code-Assisted LLMs for Mathematics

Zena Al-Khalili, Nick Howell, Dietrich Klakow · 2025

Read Evaluating Grounded Reasoning by Code-Assisted LLMs for Mathematics by Zena Al-Khalili, Nick Howell, Dietrich Klakow (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17665.txt.

Introduction

Zena Al-Khalili, Nick Howell, and Dietrich Klakow grade the reasoning inside code that chat models write for maths problems. Their 2025 paper looks past whether the code runs to whether its logic is sound. Five models were tested on several maths sets, with grading by hand and by machine.

A taxonomy sorts the programs by soundness of the reasoning they encode. Closed models grounded their code in real maths ideas. Open models leaned on memorized answers and brute-force search instead. Harder problems cut sound output for every model. The authors warn that accuracy scores flatter these systems. Soundness checks belong in future testing.

Catalog record

CREATORZena Al-Khalili, Nick Howell, Dietrich Klakow
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-17665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Grounded Reasoning by Code-Assisted LLMs for Mathematics Zena Al-Khalili, Nick Howell, Dietrich Klakow Abstract Assisting LLMs with code generation improved their performance on mathematical reasoning tasks. However, the evaluation of code-assisted LLMs is generally restricted to execution correctness, lacking a rigorous evaluation of their generated programs. In this work, we bridge this gap by conducting an in-depth analysis of code-assisted LLMs generated programs in response to math reasoning tasks. Our evaluation focuses on the extent to which LLMs ground their programs to math rules, and how that affects their end performance. For this purpose, we assess the generations of five different LLMs, on two different math datasets, both manually and automatically. Our results reveal that the distribution of grounding depends on LLMs’ capabilities and the difficulty of math problems. Furthermore, mathematical grounding is more effective for close-source models, while open-source models fail to employ math rules in their solutions correctly. On MATH500, the percentage of grounded programs decreased to half, while the ungrounded generations doubled in comparison to ASDiv grade-school problems. Our work highlights the need for in-depth evaluation beyond execution accuracy metrics, toward a better understanding of code-assisted LLMs capabilities and limits in the math domain. 1 Introduction Large Language Models (LLMs) have recently achieved outstanding performance on complex reasoning tasks such as mathematical reasoning, powered by scale and multi-step reasoning approaches, particularly Chain-of-Though (CoT) Wei et al. 2022 , that require an LLM to generate the explicit reasoning steps, before generating the final answer. Despite its success, investigating CoT reasoning steps revealed critical flows of LLMs, such as committing calculation errors Gao et al. 2023 and generating false positive chains Lyu et al. 2023 , i.e: containing reasoning errors yet generating correct final answers. Code-assisted reasoning approaches Gao et al. 2023 ; Chen et al. 2022 ; Lyu et al. 2023 ; Gou et al. 2023 ; Mickus et al. 2024 ; Das et al. 2024 proposed to solve these problems by instructing LLMs to generate programmatic reasoning steps, e.g, Python programs, and delegate the execution of these programs to an interpreter, which avoids calculation errors and ensures faithfulness. Such approaches have been found to further improve LLMs performance on math reasoning tasks. However, performance improvement is predominantly evaluated by the correctness of the execution outcome Gao et al. 2023 ; Chen et al. 2022 ; Gou et al. 2023 , rather than the quality of the generated programs and how it affects the reasoning process. This is problematic, as the generated programs can be relaying on exhaustive searches or memorized information to generate correct answers instead of grounding the reasoning process to domain concepts, i.e: mathematical rules, producing less generalizable programs that are harder to verify and trust. Figure 1 show such programs generated by GPT-4 and StarCoder2 when solving math problems from the ASDiv data. Figure 1: Programs generated by GPT-4 (left) that uses a brute force search to find the answer to the input question, and StarCoder2 (right) that depends on memorized information rather than generating the steps to solve the input question. The goal addressed in this work is to evaluate the actual reasoning capabilities of code-assisted LLMs when solving mathematical tasks by analyzing their generated intermediate programs. In this evaluation, we focus mainly on assessing the extent to which code-assisted LLMs ground their reasoning process to domain concepts and math rules when generating their solutions, and how grounding, or its absence, affects end performance on mathematical tasks. Code enables access to domain concepts such as math rules, via libraries ( e.g: Sympy), which can aid LLMs generate programs that are grounded to world knowledge ( i.e: mathematical rules), aligning more to human-like reasoning. Given this we expect grounded programs to be trustworthy, easier to verify, and probably with a higher solve rate, while ungrounded programs to be more error-prone, harder to trust, and degrade end-performance. We investigate the extent to which this is true by seeking answers to the following research questions. • RQ1 : To what extent do code-assisted LLMs ground their reasoning to relevant mathematical concepts? • RQ2 : How do code-assisted LLMs implement these concepts? • RQ3 : What is the relationship between grounding to math concepts and end performance on mathematical tasks? • RQ4 : How much do the answers to these questions vary with more challenging math problems? To answer these questions, we start by manually analyzing a subset of intermediate programs, generated by GPT4o-mini, GPT4, Qwen2.5, Llama3, and StarCoder2, for grounding and its implementation style, when solving math problems from the ASDiv and MATH500 datasets ( 3.3 ). Given the observations from the manual analysis, we design a taxonomy reflecting the different grounding levels used by evaluated models ( 3.4 ). To extend the analysis to the full set of generated programs, we employ two automated evaluation methods: an LLM-Judge using the latest o3-mini model, and our proposed method, Code-Structure Judge, that trains a Decision Tree classifier with features extracted from programs’ Abstract Syntax Tree ( 3.5 ). We find Code-Structure judge to achieve higher agreement with human judgment than LLM-judge, with 81\% and 73\% , respectively ( 5.1 ); therefore, we employ it for the large-scale evaluation of all generated programs. To the best of our knowledge this is the first work to analyze generated programs of code-assisted LLMs on math reasoning tasks. We summarize our takeaways from Section ( 5.2 ) as following: 1. Code-assisted LLMs ground their programs in math knowledge either by invoking concepts from math library or implementing the concepts from scratch. 2. Code-assisted LLMs also relay on brute-force searches or memorized information to approach some math problems. 3. Grounded generations to math knowledge are more effective for GPT models and Qwen, while open-source models fail to correctly ground their reasoning to useful concepts. 4. The grounding distribution completely changed on the challenging MATH500 data, pushing GPT models and Qwen to double their reliance on brute-force search and memorized information, at the expense of more mathematically-grounded solutions. Our in-depth evaluation highlights the need for more holistic assessment of LLMs generations, beyond execution accuracy metrics, which don’t provide insightful conclusions about models’ actual capabilities and limits in the math domain. 2 Related Work Programs as Intermediate Steps for Math Reasoning Code-assisted reasoning approaches such as Chen et al. 2022 ; Gao et al. 2023 ; Imani et al. 2023 ; Lyu et al. 2023 ; Das et al. 2024 prompt LLMs to generate programs instead of intermediate steps in natural language, which have been found to improve the performance of LLMs on math reasoning tasks. Beyond in-context learning approaches, other work Gou et al. 2023 ; Mickus et al. 2024 fine-tuned medium-scale language models such as Code-Llama Roziere et al. 2023 on code reasoning paths and achieved comparable performance to closed-source models on math reasoning tasks. However, both in-context and fine-tuned approaches are predominantly evaluated by the correctness of the final answer, overlooking the intermediate programs and how their quality might affect the reasoning process. This work focuses on exploring these programs to accurately evaluate the problem-solving abilities of code-assisted LLMs. Evaluation of Intermediate Steps This work also relates to several studies that evaluate the intermediate reasoning steps of LLMs in natural

Read the complete text