A Compute-Matched Re-Evaluation of TroVE on MATH

Tobias Sesterhenn, Ian Berlot-Attwell, Janis Zenkner, Christian Bartelt · 2025

Read A Compute-Matched Re-Evaluation of TroVE on MATH by Tobias Sesterhenn, Ian Berlot-Attwell, Janis Zenkner, Christian Bartelt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22069.txt.

Introduction

Tobias Sesterhenn and co-authors redo a test of a tool reuse method for math solving by code writing models. Their paper from 2025 covers TroVE, which splits its budget between direct code, tool making, and tool reuse on the MATH test set. TroVE claimed a clear win over a direct only base line.

Later doubts noted the tools made were often trivial or seldom reused, hinting the gain came from self checks. The retest also fixed a small bug in TroVE pick logic, worth three points. With compute held equal between the two, the TroVE edge shrank to about one point. The authors read the toolbox idea as giving no real gain on this test set.

Catalog record

CREATORTobias Sesterhenn, Ian Berlot-Attwell, Janis Zenkner, Christian Bartelt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22069

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Compute-Matched Re-Evaluation of TroVE on MATH Tobias Sesterhenn, Ian Berlot-Attwell, Janis Zenkner, Christian Bartelt Abstract Reusing established theorems and formulas is central to mathematical problem solving, serving as essential building blocks for tackling increasingly complex challenges. Recent work, TroVE , argues that code-generating Large Language Models (LLMs) can benefit similarly on the MATH benchmark by inducing and reusing higher-level toolboxes. By allocating computational budget across an ensemble of three modes – directly generating code, creating tools, and reusing tools – TroVE claims to outperform a Primitive baseline that only performs direct generation. However, recent analysis ( Berlot-Attwell et al. 2024 ) casts doubt on these gains, noting that the tools created are often trivial or rarely reused, suggesting that improvements may stem from self-consistency or self-correction. In this work, we re-evaluate TroVE on MATH , analyze the impact of each of its modes, and show that its benefit does not come from these mechanisms, but simply from a higher computational budget spent for TroVE compared to Primitive . To this end, we also perform a small correction in the original implementation of TroVE ’s selection mechanism, boosting TroVE ’s performance on MATH by 3 % 3\% in accuracy. After matching for compute, the benefit of TroVE reduces to a marginal improvement of 1 % 1\% , suggesting that this toolbox approach does not provide a significant benefit on MATH . 1 Introduction Reusing proven lemmas and formulas is how mathematicians keep cognitive load manageable and proofs short: recalling a schema (e.g., the Binomial theorem) frees working memory that would otherwise be spent on re-derivation ( Sweller 1988 ; Zhao et al. 2024 ) , and automated–reasoning systems that reuse earlier proofs solve new theorems markedly faster than systems that start from scratch ( Walther & Kolbe 2000 ) . In other words, good mathematics relies on a library of lemmas that can be imported rather than rebuilt. This reuse principle has long guided library learning in computer science. Systems such as DreamCoder ( Ellis et al. 2023 ) interleave solving tasks with compressing recurring code snippets into a growing library; the learned abstractions then make subsequent synthesis significantly faster and more sample-efficient. Later variants ( Bowers et al. 2023 ; Cao et al. 2023 ; Grand et al. 2023 ) refine the idea, but the core insight remains: discover reusable components, store them, and apply them later. Recent work has explored how to leverage In-Context Learning (ICL), where a LLM is prompted with a description of the domain, examples, a set of available functions, and the task to solve. This allows for an incrementally extended library, or “toolbox“, through a mechanism to select valid abstractions without retraining the model. These approaches ( Wang et al. 2025 ; Wang et al. 2024b ; Cai et al. 2023 ) have recently gained attention for their effectiveness in improving performance on various benchmarks. Table 1 : Accuracy comparison across MATH categories. We report mean and standard variation on 5 different random seeds. The left two columns show the improvement in performance when matching Primitive for compute. On the right we reproduce the results of TroVE and show the performance gain when changing TroVE ’s selection mechanism from a two-stage to a one-stage approach. Green highlights represent strictly higher improvements for the compute-matched version of Primitive and the corrected version of TroVE . PRIMITIVE: Reproduction TroVE: Reproduction Category Original Results Compute-Matched Original Results Reproduced Improved Algebra 0.15 0.27 \pm 0.01 0.25 0.26 \pm 0.01 0.29 \pm 0.01 Counting 0.14 0.24 \pm 0.00 0.26 0.24 \pm 0.02 0.27 \pm 0.02 Geometry 0.06 0.08 \pm 0.01 0.08 0.05 \pm 0.01 0.08 \pm 0.01 Intermediate 0.05 0.14 \pm 0.01 0.11 0.12 \pm 0.02 0.13 \pm 0.01 Number 0.16 0.28 \pm 0.02 0.25 0.27 \pm 0.01 0.30 \pm 0.01 Prealgebra 0.21 0.33 \pm 0.02 0.29 0.29 \pm 0.02 0.32 \pm 0.02 Precalculus 0.10 0.15 \pm 0.01 0.17 0.18 \pm 0.03 0.20 \pm 0.01 MATH 0.12 0.24 \pm 0.01 0.20 0.22 \pm 0.01 0.25 \pm 0.01 Among them, TroVE ( Wang et al. 2024b ) currently stands out as a state-of-the-art method on the MATH dataset ( Hendrycks et al. 2021 ) . For a fixed computational budget of K LLM calls, TroVE repeatedly generates Python code to solve a given task using three different prompting modes and then selects the most likely correct solution via self-consistency (i.e., majority vote) ( Wang et al. 2022 ) . The three modes are: • Create : Solve the task and add any new functions to a toolbox that is periodically pruned. • Import : Solve the task by using functions already present in the toolbox. • Skip : Solve the task only with primitive, built-in functions, without using the toolbox. Hence, when evaluating the use of CodeLLaMa ( Roziere et al. 2023 ) , Wang et al. 2024b reported that TroVE outperforms a Primitive baseline that operates only in SKIP mode achieving 20\% on MATH compared to Primitive only solving 12\% . However, Berlot-Attwell et al. 2024 raise the question whether the TroVE mechanism actually increases performance as they show that tools are either trivial (already contained in the knowledge of the LLM) or never reused (see Appendix B ): An ablation study in a subset of MATH shows that the removal of the Import mode does not negatively affect performance. Although the authors hypothesize TroVE ’s benefit may come from self-consistency or the self-correctness mechanism, the question of what makes TroVE better remains an open question. In this work, we investigate what really drives TroVE ’s strong performance on the MATH benchmark and show that its main advantage comes from a higher computational budget rather than the toolbox mechanism itself. We first test whether TroVE continues to outperform a Primitive baseline when both systems are given the same number of LLM calls. To ensure a fair comparison, we also fix a discrepancy between TroVE ’s described agreement-based selection method and its actual implementation, correcting an error that improves its performance by 3\% accuracy. Finally, we analyze whether TroVE ’s varied prompting strategies provide benefits independently of the toolbox, and how TroVE and Primitive scale with increased compute under an ideal selection mechanism. Our results show that while TroVE marginally outperforms Primitive , the accuracy gain reduces to only 1\% when controlling for the computational budget. We also observe that, under compute-matched settings, TroVE produces a slightly more diverse set of candidate predictions — on average 0.74 more per task. While this may offer some benefit by broadening the search space, it also increases the difficulty in selecting the correct answer. Our findings indicate that TroVE ’s improved performance is best explained by a higher allocation of the computational budget, rather than by its toolbox mechanism. 2 Revisiting TroVE Prompting Modes TroVE is a training-free method to build a toolbox of reusable high-level functions to solve programmatic tasks, operating without ground truth labels or external supervision ( Wang et al. 2024b ) . For each task, TroVE samples K candidate programs using a fixed computational budget, divided equally among three distinct prompting modes: Skip , Create , and Import . In Skip mode, the model generates a solution using only

Read the complete text