GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training

Junjie Li, Ziao Wang, NingXuan Ma, Jianghong Ma, Xiaofeng Zhang · 2026

Read GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training by Junjie Li, Ziao Wang, NingXuan Ma, Jianghong Ma, Xiaofeng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13130.txt.

Introduction

Li and four co-authors present GRACE, a gradient-aligned method for curating reasoning data. Existing pipelines score whole samples, treating every step as equally valuable, yet steps contribute very unevenly. GRACE scores each step by its alignment with the answer gradient and its consistency with the preceding trajectory, aggregating step scores into a sample value for subset selection. A representation-level gradient proxy estimates alignment in a single forward pass. Training Qwen3-VL-2B-Instruct on MMathCoT-1M, GRACE hits 108.8% of full-data performance with 20% of the data.

Catalog record

CREATORJunjie Li, Ziao Wang, NingXuan Ma, Jianghong Ma, Xiaofeng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53871
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training Junjie Li, Ziao Wang, NingXuan Ma, Jianghong Ma, Xiaofeng Zhang Abstract Existing reasoning data curation pipelines score whole samples, treating every intermediate step as equally valuable. In reality, steps within a trace contribute very unevenly, and selecting reasoning data well requires assessing them individually. We present GRACE, a gradient-aligned curation method that views each reasoning trace as a sequence of optimization events and scores every step by two complementary signals: its alignment with the answer-oriented gradient direction, and its consistency with the preceding reasoning trajectory. Step-level scores are aggregated into a sample-level value for subset selection, using only the model’s internal optimization signals and no external reward models or step annotations. To make this scalable, GRACE introduces a representation-level gradient proxy that estimates step-level alignment from token-level upstream signals in a single forward pass. Post-training Qwen3-VL-2B-Instruct on MMathCoT-1M, GRACE reaches 108.8% of the full-data performance with 20% of the data and retains 100.2% with only 5%, with subsets that transfer effectively across model backbones. 1 Introduction Large-scale reasoning datasets have become a cornerstone for post-training large language and vision-language models [ 30 , 35 ] . The standard way to use them is to supervise the model on the entire reasoning trace, treating every step as an equally valuable target. In reality, the steps within a trace contribute very unevenly: some directly support the final answer, while others restate earlier content, explore irrelevant tangents, or introduce noise. Training on them uniformly wastes budget on low-value steps and dilutes the contribution of useful ones. This cost has become significant as reasoning corpora grow to millions of traces [ 19 ] and post-training takes hundreds of GPU-hours per run. Choosing what to train on therefore matters [ 39 ] , and for reasoning data this means assessing steps individually rather than ranking whole traces. Existing data curation methods improve training efficiency by selecting samples based on correctness [ 29 ] , reward models [ 20 ] , or sample-level influence [ 32 , 16 , 10 ] and all of them operate at the granularity of entire traces. As a result, a trace with correct final answers but poor intermediate steps [ 27 ] is treated as equally valuable as a tightly reasoned one. This highlights a fundamental limitation: current approaches lack a mechanism to assess how individual reasoning steps contribute to optimization . Figure 1: Motivation and empirical effect of GRACE. Left: Reasoning traces are viewed as sequences of optimization events, where each step induces an update direction whose utility depends on its alignment with the target objective and the evolving trajectory. Right: Radar chart comparing downstream performance across benchmarks. GRACE achieves full-data or better performance using only a fraction of the training data. In this work, we revisit reasoning data from an optimization perspective. Instead of viewing reasoning traces as static supervision targets, we model them as sequences of optimization events, where each reasoning step induces a local training signal that affects the gradient direction toward the final answer. From this view, the utility of reasoning data depends not only on external attributes such as correctness or length, but also on whether its intermediate steps constructively support optimization. Building on this view, we propose GRACE ( G radient-aligned R easoning d A ta C uration for E fficient post-training), a method that performs fine-grained data curation by estimating step-level optimization utility. Rather than pruning or rewriting reasoning traces, GRACE assigns each step a utility score based on two complementary criteria: (i) its alignment with the answer-oriented optimization direction, and (ii) its consistency with the accumulated reasoning trajectory. These signals capture both task-driven and trajectory-aware contributions of each step. The resulting step-level scores are then aggregated to produce a sample-level utility score, enabling effective subset selection while preserving the simplicity of sample-level training. Fig. 1 illustrates the motivation of GRACE and provides empirical evidence that optimization-aware curation can retain strong performance with substantially fewer training samples. A key challenge is that true step-level gradients are computationally intractable at scale. Naively computing gradients for each reasoning step would require decomposing traces into multiple training instances and performing repeated backward passes. To address this, GRACE introduces a representation-level gradient proxy that approximates step-induced optimization directions using token-level upstream signals. This proxy enables efficient estimation of step-level alignment from a single forward pass, making optimization-aware curation practical for large-scale CoT datasets. We evaluate GRACE by post-training Qwen3-VL-2B-Instruct [ 26 ] on MMathCoT-1M [ 19 ] and assessing the resulting models on a diverse suite of multimodal benchmarks spanning mathematical reasoning and general visual question answering. GRACE consistently identifies high-value subsets: training on only 20% of the curated data surpasses full-data performance, reaching 108.8% of the full-data result averaged across benchmarks, while using only 5% retains 100.2% . Furthermore, the selected subsets transfer effectively across model backbones, suggesting that the proposed optimization-based signal captures intrinsic data value beyond a specific model configuration. Our contributions are three-fold: 1. We introduce an optimization-based perspective on reasoning data, framing reasoning traces as sequences of optimization events and highlighting the role of step-level alignment in effective learning. 2. We propose GRACE, a reasoning data curation method that aggregates step-level optimization signals derived from answer-oriented alignment and trajectory consistency for sample-level subset selection. 3. We develop a representation-level gradient proxy that enables scalable estimation of step-level alignment without per-step parameter-space gradient computation. 2 Method Figure 2: The GRACE curation pipeline. (1) Given an input and its reasoning trace, GRACE identifies token sets for each step and the answer. (2) A fixed scoring model extracts token-level upstream signals in one forward pass and groups them by token sets. (3) Grouped signals are averaged into gradient proxies and scored by answer and trajectory alignment. (4) Step scores are aggregated into a sample value for ranking and top- \rho subset selection. In this section, we present GRACE. We first define the reasoning data curation problem, then derive the step-level optimization utility, introduce its scalable representation-level proxy, and describe sample-level subset selection. The overall pipeline is illustrated in Fig. 2 . 2.1 Problem Formulation We consider a reasoning dataset \mathcal{D}=\{z_{i}\}_{i=1}^{N} , where each sample is z_{i}=(x_{i},\mathbf{s}_{i},a_{i}), with x_{i} denoting the input, \mathbf{s}_{i}=(s_{i,1},s_{i,2},\dots,s_{i,K_{i}}) denoting a sequence of reasoning steps, and a_{i} denoting the final answer. Let f_{\theta} denote the model, and let \mathcal{T}_{i,k} and \mathcal{T}_{i,\mathrm{ans}} denote the token positions of step s_{i,k} and the answer segment, respectively. For any token set \mathcal{T} , we define the average token-level loss L(\theta;z_{i},\mathcal{T})=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}L_{t}(\theta;z_{i}). Accordingly, the step loss and answer loss are given by L_{i,k}=L(\theta;z_{i},\mathcal{T}_{i,k}) and L_{i}^{\mathrm{ans}}=L(\theta;z_{i},\mathcal{T}_{i,\mathrm{ans}}) , and the full loss over the reasoning trace and answer is \mathcal{T}_{i,\mathrm{full}}=\bigcup_{k=1}^{K_{i}}\mathcal{T}_{i,k}\cup\mathcal{T}_{i,\mathrm{ans}},\qquad L_{\mathrm{full}}(\theta;z_{i})=L(\theta;z_{i},\mathcal{T}_{i,\mathrm{full}}). Standard post-training minimizes \mathcal{L}_{\mathrm{SFT}}(\theta;\mathcal{D})=\frac{1}{|\mathcal{D}|}\sum_{z_{i}\in\mathcal{D}}L_{\mathrm{full}}(\theta;z_{i}). (1) Our goal is to select a compact subset \mathcal{S}\subset\mathcal{D} with budget |\mathcal{S}|=\lceil\rho|\mathcal{D}|\rceil , where \rho\in(0,1) is the selection ratio, such that post-training on \mathcal{S} preserves or improves downstream performance compared with training on the full dataset. To this end, GRACE assigns each sample a scalar value score V(z_{i}) and selects the top-ranked subset: \mathcal{S}=\left\{z_{i}\in\mathcal{D}\mid\operatorname{rank}_{V}(z_{i})\leq\lceil\rho|\mathcal{D}|\rceil\right\}. (2) The key question is how to define V(z_{i}) for reasoning data. GRACE addresses this by estimating the optimization utility of each reasoning step and aggregating these step-level signals into a

Read the complete text