AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
Zihan Liu †, Yang Chen †, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping † · 2024
Read AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling by Zihan Liu †, Yang Chen †, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping † (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15084.txt.
Introduction
Zihan Liu and co-authors introduce AceMath, a family of math-focused language models, in this 2024 paper. The suite pairs solver models with reward models that grade candidate answers and pick the right ones. Training ran in two stages: broad fine-tuning first, then a targeted stage with curated math prompts and machine-made answers.
Their largest solver beat strong rivals, including well-known commercial systems, on math tests. The reward model also led its class on a new grading benchmark the team built. Joined together, the pair posted the best average score across the math suites tested. The team released the weights, training data, and benchmarks for public use.
Catalog record
| CREATOR | Zihan Liu †, Yang Chen †, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping † |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15084
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling Zihan Liu †, Yang Chen †, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping † Abstract In this paper, we introduce AceMath, a suite of frontier math models that excel in solving complex math problems, along with highly effective reward models capable of evaluating generated solutions and reliably identifying the correct ones. To develop the instruction-tuned math models, we propose a supervised fine-tuning (SFT) process that first achieves competitive performance across general domains, followed by targeted fine-tuning for the math domain using a carefully curated set of prompts and synthetically generated responses. The resulting model, AceMath-72B-Instruct greatly outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet. To develop math-specialized reward model, we first construct AceMath-RewardBench , a comprehensive and robust benchmark for evaluating math reward models across diverse problems and difficulty levels. After that, we present a systematic approach to build our math reward models. The resulting model, AceMath-72B-RM , consistently outperforms state-of-the-art reward models. Furthermore, when combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest average rm@8 score across the math reasoning benchmarks. We release model weights, training data, and evaluation benchmarks at: https://research.nvidia.com/labs/adlr/acemath . † † affiliationnotice: Equal contribution 1 Introduction Figure 1: AceMath versus leading open-weights and proprietary LLMs on math reasoning benchmarks. Additionally, we report rm@8 accuracy (best of 8) with our reward model AceMath-72B-RM and use the official reported numbers from Qwen2.5-Math. Over the past year, the open large language model (LLM) community has made remarkable progress in advancing the key capabilities of LLMs, including multi-turn conversation ( Chiang et al., 2023 ; Dubey et al., 2024 ) , coding ( Guo et al., 2024 ; Hui et al., 2024 ) , multimodal functionalities ( Dai et al., 2024 ; Chen et al., 2024 ) , retrieval-augmented generation (RAG) ( Liu et al., 2024c ) , and mathematical reasoning ( Azerbayev et al., 2023 ; Shao et al., 2024 ; Mistral, 2024 ; Yang et al., 2024b ) . Among these capabilities, mathematics is recognized as a fundamental aspect of intelligence. It can serve as a reliable benchmark due to its objective, consistent, verifiable nature. Consequently, solving math problems is widely regarded as a critical testbed for evaluating an LLM’s ability to tackle challenging tasks that require complex, numerical and multi-step logical reasoning ( Cobbe et al., 2021 ; Hendrycks et al., 2021a ; Lightman et al., 2023 , e.g.,) . Previous studies have convincingly demonstrated that math-specialized LLMs significantly outperform general-purpose LLMs on challenging mathematical benchmarks ( Azerbayev et al., 2023 ; Shao et al., 2024 ; Mistral, 2024 ; Yang et al., 2024b ) . These math-specialized models, including the corresponding reward models (a.k.a. verifiers), are not only valuable to the mathematics and science communities ( Tao, 2023 , e.g.,) , but they also provide valuable insights into data collection and serve as synthetic data generation tools, contributing to the advancement of future iterations of general-purpose LLMs. The improved mathematical reasoning capabilities of math-specialized LLMs are generally acquired through both the continued pre-training and post-training: i) During continued pre-training stage, the models are initialized with general-purpose base pretrained LLMs (e.g., Llama-3.1-70B ( Dubey et al., 2024 ) ), and continually trained on extensive collections of mathematical corpora, often comprising hundreds of billions of tokens sourced from Common Crawl ( Shao et al., 2024 ) , ArXiv papers ( Azerbayev et al., 2023 ) , and synthetically generated datasets ( Yang et al., 2024b ; Akter et al., 2024 ) . In this stage, losses are calculated on every token within the corpus. ii) In the post-training phase, the continually pretrained math base LLMs (e.g., Qwen2.5-Math-72B ( Yang et al., 2024b ) ) are fine-tuned using large datasets of mathematical prompt-response pairs. In this stage, losses are computed only on the response tokens, allowing the models to refine their ability to generate accurate answers given the prompts or problem descriptions. In this work, we push the limits of math reasoning with post-training and reward modeling based on open weights base LLMs and math base LLMs. We establish state-of-the-art supervised fine-tuning (SFT) and reward modeling (RM) processes for building math-specialized models, while also sharing key insights gained from our comprehensive studies. Specifically, we make the following contributions: 1. We introduce a SFT process designed to first achieve competitive performance across general domains, including multidisciplinary topics, coding, and math. Building on this, the general SFT model is further fine-tuned in math domain using a meticulously curated set of prompts and synthetically generated responses. Leveraging the high-quality training data, the resulting model, AceMath-7B-Instruct, largely outperforms the previous best-in-class Qwen2.5-Math-7B-Instruct (pass@1: 67.2 vs. 62.9) on a variety of math reasoning benchmarks (detailed results in Figure 1 ), while coming close to the performance of 10 \times larger Qwen2.5-Math-72B-Instruct (67.2 vs. 68.2). Notably, our AceMath-72B-Instruct outperforms the state-of-the-art Qwen2.5-Math-72B-Instruct by a margin (71.8 vs. 68.2). 2. We conducted a systematic investigation of training techniques for building math-specialized reward models, focusing on key aspects such as the construction of positive-negative pairs, training objectives, and the elimination of stylistic biases from specific LLMs. Leveraging the insights gained from this exploration, our AceMath-72B-RM consistently outperforms state-of-the-art reward models, including Qwen2.5-Math-RM-72B and Skywork-o1-Open-PRM-Qwen-2.5-7B ( Skywork-o1, 2024 ) , in the math domain. Moreover, when combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest average rm@8 score across seven math reasoning benchmarks (see Figure 1 ), setting a new standard for performance in this field. 3. We will open source the model weights for AceMath-Instruct and AceMath-RM, along with the complete training data used across all stages of their development. We also release AceMath-RewardBench, a comprehensive benchmark for evaluating math reward models, offering diverse datasets, varying difficulty levels, and robustness to variations in response styles. We organize the rest of this paper as follows. In § 2 , we introduce the related work. In § 3 , we introduce the details on the SFT training data curation. We present our reward model training in § 4 . We conclude the paper in § 5 . 2 Related Work 2.1 Continued Pre-training on Math Corpus Many studies have investigated the integration of large-scale mathematical data for pre-training LLMs to enhance their math capabilities ( Shen et al., 2021 ; Wang et al., 2023 ; Zhang et al., 2024a ; Ying et al., 2024 ; Akter et al., 2024 ; Hui et al., 2024 ) . Additionally, some research has focused on developing math-specialized LLMs by continuing the pre-training of a general-purpose LLM with an extensive math corpus, sourced from math-related web texts, encyclopedias, exam questions, and synthetic mathematical data ( Shao et al., 2024 ; Yang et al., 2024b ) . These works demonstrate that this additional math-focused pre-training significantly enhances the model’s ability to solve math problems, benefiting not only the pre-trained base model but also subsequent instruct models after post-training. 2.2 Supervised Fine-Tuning Numerous supervised fine-tuning (SFT) datasets have been developed to enhance pretrained LLMs with versatile capability, such as instruction following ( Chiang et al., 2023 ; The-Vicuna-Team, 2023 ; Lian et al., 2023 ; Mukherjee et al., 2023 ; Teknium, 2023 ; Peng et al., 2023 ; Yuan et al., 2024 ) , coding ( Glaive-AI, 2023 ;