ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations

Quang Hieu Pham, Thuy Duong Nguyen, Tung Pham, Anh Tuan Luu, Dat Quoc Nguyen · 2025

Read ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations by Quang Hieu Pham, Thuy Duong Nguyen, Tung Pham, Anh Tuan Luu, Dat Quoc Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03763.txt.

Introduction

The authors teach language models math by having them fill in missing equations, like a cloze exercise in school. Standard training predicts the next word, which may not mirror how people learn to reason. In their setup, key equations in a worked solution are masked, and the model learns to restore them.

On three math test sets, the method beat a strong masked-thought baseline in both score and robustness. Two test-time decoding schemes added further gains. A breakdown study shows which design choices drive the improvement, from masking style to model layout.

Catalog record

CREATORQuang Hieu Pham, Thuy Duong Nguyen, Tung Pham, Anh Tuan Luu, Dat Quoc Nguyen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03763

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations Thanks: This work was completed while all authors, except the fourth, were at Movian AI, Vietnam. All datasets and models were downloaded, trained, and evaluated using Movian AI’s resources. Thanks: Qualcomm Vietnam Company Limited. Quang Hieu Pham, Thuy Duong Nguyen, Tung Pham, Anh Tuan Luu, Dat Quoc Nguyen Abstract The capabilities of large language models (LLMs) have been enhanced by training on data that reflects human thought processes, such as the Chain-of-Thought format. However, evidence suggests that the conventional scheme of next-word prediction may not fully capture how humans learn to think. Inspired by how humans generalize mathematical reasoning, we propose a new approach named ClozeMath to fine-tune LLMs for mathematical reasoning. Our ClozeMath involves a text-infilling task that predicts masked equations from a given solution, analogous to cloze exercises used in human learning. Experiments on GSM8K, MATH, and GSM-Symbolic show that ClozeMath surpasses the strong baseline Masked Thought in performance and robustness, with two test-time scaling decoding algorithms, Beam Search and Chain-of-Thought decoding. Additionally, we conduct an ablation study to analyze the effects of various architectural and implementation choices on our approach. 1 Introduction Figure 1: ClozeMath’s mixture of training objectives. In text infilling, masked equation spans are replaced with special mask tokens (<X>, <Y>) and are gradually predicted from left to right. These mask tokens ensure alignment of token positions during training. A special <SEP> token separates the input and target parts, using bidirectional and causal attention masks, respectively. To mimic human reasoning in mathematical settings, current Large Language Models (LLMs) are prompted or trained to generate intermediate thinking steps before reaching a conclusion Wei et al. 2022 ; Kojima et al. 2022 . Some efforts focus on generating data in this format by collecting responses from humans or LLMs Lightman et al. 2024 ; Yue et al. 2024 or by implicitly learning task-specific representations Wang et al. 2024 . However, while these outputs resemble the surface form of human thought, the current language modeling objective may not align with how humans learn to think Bachmann and Nagarajan 2024 ; Bender et al. 2021 . When learning from a mathematical solution, it is easier for a human to generalize the main approach before working on the details, rather than memorizing which step follows another Mason et al. 2010 . Considering the input problem in Figure 1 , the textual part of the target solution describes its general approach (i.e., rationales), while the equations are problem-specific. In light of this observation, we propose enhancing the training of language models for mathematical reasoning by introducing an additional text infilling objective Raffel et al. 2020 ; Tay et al. 2023b ; Tay et al. 2023a . In this approach, models learn to predict masked equations alongside the standard language modeling objective. This method reinforces the model’s ability to infer mathematical relationships by predicting equations based on the surrounding rationale, encouraging a more structured understanding of problem-solving steps. While traditional text infilling objectives involve masking and predicting random text spans, we find that this approach disrupts the logical coherence of the unmasked parts available to the model, ultimately harming reasoning performance. Regarding model architecture, PrefixLM Liu et al. 2018 , which applies a bidirectional attention mask to the model’s prompt, has been shown to work well with text infilling in T5 Raffel et al. 2020 . However, in the context of mathematical reasoning, we demonstrate that only our equation masking strategy benefits from this architectural choice. From a different perspective, Masked Thought Chen et al. 2024 , a recent method with strong empirical results, regularizes a model to look further back into a math problem’s definition by randomly corrupting the problem’s solution during language model training. However, mathematical solutions may include auxiliary derived steps that are not directly connected to the problem definition. We demonstrate a common example in which Masked Thought may teach the model spurious correlations in Section 3.2 . Empirically, our method also substantially outperforms Masked Thought. In summary, our contributions are as follows: (1) We introduce a simple yet highly effective fine-tuning strategy, namely ClozeMath, that enhances the mathematical reasoning abilities of LLMs. (2) We conduct extensive experiments on two standard benchmarks, GSM8K Cobbe et al. 2021 and MATH Hendrycks et al. 2021 , including a study on ClozeMath scalability on test-time compute, and perform robustness analyses on the newly released benchmark GSM-Symbolic Mirzadeh et al. 2025 . We also show that ClozeMath outperforms the recent strong baseline Masked Thought. (3) We present a comprehensive ablation study that explores the impact of various architectural choices on the effectiveness of our proposed method. 2 Our ClozeMath approach Let \{X^{i},Y^{i}\}_{i=1}^{n} be a training set, in which X^{i} is a text-based mathematical problem (e.g., X^{i} is the Input in the Language modeling part in Figure 1 ), and Y^{i}=y^{i}_{1}\ y^{i}_{2}\ \dots\ y^{i}_{m_{i}} , where each y^{i}_{m} is a text segment or a mathematical equation (e.g., Y^{i} is the Target in the Language modeling part in Figure 1 ; then m_{i}=5 and y^{i}_{1} is "Natalia sold", y^{i}_{2} is "48/2 = 24", y^{i}_{3} is "clips in May. Natalia sold", y^{i}_{4} is "48+24 = 72" and y^{i}_{5} is "clips altogether in April and May."). The language modeling objective is formulated as: \displaystyle\mathcal{L}_{\mathrm{lm}}= \displaystyle\sum_{i=1}^{n}P(Y^{i}|X^{i}) (1) Let F^{i}=\{f_{1}^{i},f^{i}_{2},\dots\,f^{i}_{|F^{i}|}\}\subset\{1,2,...,m_{i}\} be the set of indices such that \forall m\in F^{i} , y_{m}^{i} is a mathematical equation (e.g., from the example in the paragraph above, F^{i}=\{2,4\} ). Our text-infilling objective is formulated as follows: \displaystyle\mathcal{L}_{\mathrm{tf}}= \displaystyle\sum_{i=1}^{n}\sum_{\mathrm{M}\in\mathcal{M}(F^{i})}P(y^{i}_{m\in\mathrm{M}}|X^{i},Y^{i}\setminus y^{i}_{m\in\mathrm{M}}) (2) where \mathcal{M}(F^{i})=\{\{f^{i}_{t:|F^{i}|}\}\mid t=1,\dots,|F^{i}|\} contains combinations of equations to be masked (e.g., given F^{i}=\{2,4\} , we have \mathcal{M}(F^{i})=\{\{2,4\},\{4\}\} ). This involves masking all equations and then gradually unmasking them from the first to the last, as each equation only involves computations from its previous counterparts. For groups of simultaneous mathematical transformations, such as systems of equations, we randomly mask 50% of each group. Finally, our ClozeMath approach trains to minimize the two objectives simultaneously: \displaystyle\mathcal{L}_{\text{ClozeMath}}=\mathcal{L}_{\mathrm{lm}}+\mathcal{L}_{\mathrm{tf}} (3) Since the number of terms in \mathcal{L}_{\mathrm{tf}} depends on the number of equations to be masked within each dataset, in practice, we balance the two training objectives by duplicating samples optimized for \mathcal{L}_{\mathrm{lm}} to maintain a roughly 50:50 sample ratio. We implement ClozeMath with PrefixLM Liu et al. 2018 on pre-trained decoder-only models. In PrefixLM, the prompt part’s attention is bidirectional, while the target sequence uses a causal mask. To let the model learn to use the two attention patterns correctly, there is a special <SEP> token separating the two parts. During inference, we employ the conventional next-token prediction. 3 Experiments Figure 2: Implementation of MFT Chen et al. 2024 and ClozeMath on the same sample. Here, *** denotes the unshown causal attention mask in this figure. MFT aims at learning longer-range dependencies by randomly inputting noisy mask (<M>) tokens into the solution’s parts while keeping the problem definition untouched. MFT is prone to learning spurious correlation; in this example, a derived transformation might not fully perceive its predecessors. 3.1 Experimental setup Datasets: We conduct experiments on two standard benchmarks for mathematical problem solving, GSM8K Cobbe et al. 2021 and MATH Hendrycks et al. 2021 . To demonstrate the context-understanding ability of our method,

Read the complete text