To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization
Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, Fangzhen Lin, Hong Kong University of Science, Technology, INFLY Tech · 2025
Read To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization by Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, Fangzhen Lin, Hong Kong University of Science, Technology, INFLY Tech (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00691.txt.
Introduction
Haozhe Wang and co-authors teach a math language model to decide for itself when to write code, in work from 2025. Mixing chain-of-thought reasoning with executed code beats either alone, but fixed templates cannot adapt as the model improves.
Reinforcement learning alone explores the huge space of reasoning-code blends too slowly. The proposed framework alternates two steps. An expectation step explores structured blends of words and code. A maximization step then trains on the best patterns with off-policy reinforcement learning. The cycle sharpened both decisions and skill together. A seven-billion-parameter model gained over eleven points on one benchmark and about nine on a harder contest set, without long reasoning traces.
Catalog record
| CREATOR | Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, Fangzhen Lin, Hong Kong University of Science, Technology, INFLY Tech |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00691
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization Haozhe Wang, Long Li, Chao Qu, Fengming Zhu, Weidi Xu, Wei Chu, Fangzhen Lin, Hong Kong University of Science, Technology, INFLY Tech Abstract Recent advances in mathematical problem-solving with language models (LMs) integrate chain-of-thought (CoT) reasoning and code execution to harness their complementary strengths. However, existing hybrid frameworks exhibit a critical limitation: they depend on externally dictated instructions or rigid code-integration templates, lacking metacognitive awareness – the capacity to dynamically evaluate intrinsic capabilities and autonomously determine when and how to integrate tools. This rigidity motivates our study of autonomous code integration, enabling models to adapt tool-usage strategies as their reasoning abilities evolve during training. While reinforcement learning (RL) shows promise for boosting LLM reasoning at scale (e.g., DeepSeek-R1), we demonstrate its inefficiency in learning autonomous code integration due to inadequate exploration of the vast combinatorial space of CoT-code interleaving patterns. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes structured exploration (E-step) with off-policy RL optimization (M-step), creating a self-reinforcing cycle between metacognitive tool-use decisions and evolving capabilities. Experiments reveal our method achieves superior results through improved exploration. Notably, our 7B model improves over 11% on MATH500 and 9.4% on AIME without o1-like CoT. Code, models and data are released on https://github.com/HaozheH3/AutoCode . 1 Introduction Large Language Models (LLMs) have demonstrated remarkable performance across various domains ( Kaddour et al., 2023 ; Achiam et al., 2023 ; Dubey et al., 2024 ; Team et al., 2023 ; Yang et al., 2024a ) . Yet, solving complex mathematical problems still remains challenging, as the task requires hybrid skills in abstract reasoning, symbolic manipulation, and precise numerical computation ( Gao et al., 2023 ; Yue et al., 2023 ; Gou et al., 2023 ; Li, 2024 ) . Current approaches adopt two complementary paradigms: (1) chain-of-thought (CoT) reasoning , which decomposes problems into intermediate reasoning steps ( Wei et al., 2022 ; Yu et al., 2023 ) , and (2) external tool integration , where models generate code snippets to offload computations to interpreters or symbolic solvers ( Toshniwal et al., 2024 ; Yue et al., 2023 ) . While CoT reasoning excels at semantic parsing and stepwise logic, its reliance on token-level autoregressive generation often propagates numerical errors. Conversely, tool-based approaches ensure computational precision but suffer from a semantic-to-symbolic translation gap, where even minor syntactic errors or contextual misinterpretations disrupt execution ( Li, 2024 ) . Figure 1: Analysis of existing tool-integrated math LLMs. While tool integration can in principle bring complementary benefits to CoT reasoning, existing models show critical rigidity when incorporating code, lacking the metacognitive capacity to earn the synergistic benefits from code integration. Recent hybrid frameworks like Mammoth ( Yue et al., 2023 ) , Deepseek-Math ( Gou et al., 2023 ; Shao et al., 2024 ) , and Qwen-2.5-Math ( Yang et al., 2024b ) attempt to combine these paradigms through interleaved CoT-code reasoning. However, as our empirical analysis reveals (Fig. 1 ), current methods exhibit a critical rigidity: they either default to CoT reasoning unless explicitly prompted for code generation or adhere to static templates for tool invocation. We trace this limitation to prevailing supervised fine-tuning (SFT) paradigms that condition models to (1) passively follow user instructions (e.g., "Let’s write a Python program" ( Yue et al., 2023 ) ), (2) replicate fixed code-integration patterns from curated datasets ( Yang et al., 2024b ) , or (3) imitate teacher-forced tool-use trajectories ( Gou et al., 2023 ; Shao et al., 2024 ) . Consequently, LLMs lack metacognitive awareness – the capacity to dynamically evaluate their intrinsic capabilities against problem contexts and autonomously determine when and how to integrate tools. This deficiency motivates our central research question: How can mathematical LLMs learn autonomous code integration (AutoCode) that optimally complements their inherent reasoning capabilities? Reinforcement learning (RL) offers a promising pathway by optimizing policies through self-generated trajectories, as evidenced by recent successes like DeepSeek R1 ( Guo et al., 2025 ) . However, we empirically observe that standard RL methods is inefficient in learning autonomous code integration ( AutoCode ) strategies (see Sec. 4.2 ). This stems from RL’s tendency to exploit local policy neighborhoods, thereby insufficiently exploring the vast combinatorial space of potential CoT-code interleaving patterns. Such myopic exploration constrains the discovery of high-reward reasoning paths that judiciously blend both modalities, particularly as the model’s reasoning capabilities evolve during training. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes guided exploration with policy optimization. Our key innovation lies in formulating code-integration decisions as latent variables within an EM paradigm, creating a self-reinforcing cycle: the E-step identifies high-potential code-integration decisions through guided exploration, while the M-step optimizes policy parameters for joint metacognitive tool-usage and reasoning. This dual mechanism enables models to adapt tool-use strategies as their capabilities evolve during training. Practically, we achieve efficiency through two design choices: (1) an offline data curation step (E-step) that prioritizes high-return code invocation decisions through guided exploration, and (2) an off-policy RL optimization step (M-step) that jointly improves tool-usage and reasoning. This approach offers enhanced control and efficiency compared to standard RL, which is particularly beneficial for resource-constrained companies or researchers. Extensive experiments demonstrate that our method (a) preserves higher training efficiency while achieving better performance, and (b) learns intelligent code integration strategies that achieves higher accuracy than either CoT or code prompted in isolation. Notably, our show consistent improvements across different benchmarks, raising MATH500 from 60.4% to 71.4%. Our contribution is summarized as follows: (1) We diagnose a critical gap in mathematical LLM – the inability to autonomously integrate tools based on metacognitive awareness – and demonstrate standard RL’s inefficiency in addressing it. (2) We propose a novel EM-based framework that jointly adapts the tool-usage strategies with evolving reasoning abilities, with a simple yet efficient implementation. (3) We demonstrate superior results in both training efficiency and accuracy on challenging benchmarks. 2 Background Figure 2: Example of Autonomous Code Integration. We aim to enable LLMs to determine tool-usage strategies based on their own capability boundaries. In the example, the model write code to solve the problem that demand special tricks, strategically bypassing its inherent limitations. Problem Statement. Modern tool-augmented language models address mathematical problems x_{q}\in\mathcal{X}_{Q} by generating step-by-step solutions that interleave natural language reasoning with executable Python code (Fig. 2 ). Formally, given a problem x_{q} , a model \mathcal{M}_{\theta} iteratively constructs a solution y_{a}=\{y_{1},\dots,y_{T}\} by sampling components y_{t}\sim p(y_{t}|y_{<t},x_{q}) , where y_{<t} encompasses both prior reasoning steps, code snippets and execution results \mathbf{e}_{t} from a Python interpreter. The process terminates upon generating an end token, and the solution is evaluated via a binary reward r(y_{a},x_{q})=\mathbb{I}(y_{a}\equiv y^{*}) indicating equivalence to the ground truth y^{*} . The learning objective is formulated as: \max_{\theta}\mathbb{E}_{x_{q}\sim\mathcal{X}_{Q}}\left[r(y_{a},x_{q})\right] Challenge and Motivation. Developing autonomous code integration (AutoCode) strategies poses unique challenges, as optimal tool-usage behaviors must dynamically adapt to a model’s intrinsic capabilities and problem-solving contexts. While traditional supervised fine-tuning (SFT) relies on imitation learning from expert demonstrations, this paradigm fundamentally limits the emergence of self-directed tool-usage strategies. Unfortunately, current math LLMs predominantly employ SFT to orchestrate tool integration ( Yue et al., 2023 ;