Unified Data Selection for LLM Reasoning

Xiaoyuan Li, Yubo Ma, Chengpeng Li, Fengbin Zhu, Yiyao Yu, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu · 2026

Read Unified Data Selection for LLM Reasoning by Xiaoyuan Li, Yubo Ma, Chengpeng Li, Fengbin Zhu, Yiyao Yu, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22389.txt.

Introduction

Li and colleagues propose High-Entropy Sum, a training-free metric for judging language model reasoning. It sums only the entropy of the top 0.5 percent highest-entropy tokens in each reasoning sample. Training on the top 20 percent of HES data matches full-dataset performance in supervised fine-tuning, while the lowest data degrades it. In rejection fine-tuning, HES-based training beats baselines. In reinforcement learning, HES-picked successful trajectories teach strong reasoning patterns. The metric thus unifies data selection across three paradigms at far lower compute cost.

Catalog record

CREATORXiaoyuan Li, Yubo Ma, Chengpeng Li, Fengbin Zhu, Yiyao Yu, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60018
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22389

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Data Selection for LLM Reasoning Xiaoyuan Li, Yubo Ma, Chengpeng Li, Fengbin Zhu, Yiyao Yu, Keqin Bao, Wenjie Wang, Fuli Feng, Dayiheng Liu Abstract Effectively training Large Language Models (LLMs) for complex, long-CoT reasoning is often bottlenecked by the need for massive high-quality reasoning data. Existing methods are either computationally expensive or fail to reliably distinguish high- from low-quality reasoning samples. To address this, we propose High-Entropy Sum (HES) , a training-free metric that quantifies reasoning quality by summing only the entropy of the top ( e.g. , 0.5%) highest-entropy tokens in each reasoning sample. We validate HES across three mainstream training paradigms: Supervised Fine- tuning (SFT), Rejection Fine-tuning (RFT), and Reinforcement Learning (RL), with extensive results demonstrating its consistent effectiveness and significantly reduced computational overhead. In SFT, training on the top 20% HES-ranked data matches full-dataset performance, while using the lowest-HES data degrades it. In RFT, our HES-based training approach significantly outperforms baseline methods. In RL, HES-selected successful trajectories enable the model to learn strong reasoning patterns, significantly surpassing other compared methods. Our findings establish HES as a robust, training-free metric that enables a unified, effective, and efficient method for developing advanced reasoning in LLMs. 1 Introduction The ability of Large Language Models (LLMs) to solve complex problems through Chain-of-Thoughts (CoT) reasoning has become a central focus in frontier research ( 21 ; 11 ; 54 ) . To enhance models’ reasoning capabilities, dominant training paradigms, including Supervised Fine-tuning (SFT) ( 32 ) , Rejection Fine-tuning (RFT) ( 57 ) , and Reinforcement Learning (RL) ( 42 ) , heavily rely on high-quality training data. However, the indiscriminate expansion of training data often introduces more noise and incurs additional costs, spurring a critical need for efficient and effective data selection methods ( 60 ) . The key lies in defining a robust metric to rapidly and accurately distinguish high-quality data from low-quality data ( 25 ) . Figure 1: The comparative analysis of discriminative ability is conducted on 512 responses per problem generated by Qwen3-14B for AIME 2025. The results demonstrate that metric of (d) Sum Entropy of High-Entropy Tokens is the most effective to distinguish between high- and low-quality samples. Current research has explored various strategies for filtering training data, such as filtering by length ( 34 ) , perplexity ( 29 ) , or average token entropy ( 38 ) . However, these metrics share common limitations when applied to long-CoT reasoning scenarios, where extended and methodical reasoning processes are particularly valuable for model learning ( 8 ) . Specifically, they rely on a coarse-grained, global evaluation of the reasoning path, assigning equal weight to all tokens. This uniform averaging fails to capture the nuanced, multi-stage structure of complex reasoning, which often involves planning, exploration, and reflection ( 8 ) . Although some methods address this by training auxiliary task-specific models for data selection ( 44 ) or by employing powerful LLMs to choose among multiple responses ( 47 ) , such approaches incur substantial computational overhead . Moreover, because these methods are not directly aligned with or adaptive to the target model being optimized, they risk yielding suboptimal training data. To address the above gaps, our objective is to develop an efficient method for selecting high-quality reasoning samples by identifying and leveraging fine-grained, critical information embedded in long-CoT responses. Recently, 48 has suggested that a small number of high-entropy critical tokens in the reasoning process serve as a key driver for performance improvement. Inspired by this, we conduct a preliminary study on token-level entropy distribution throughout the reasoning process. Note that high-entropy tokens are identified as those falling within the top 0.5% of the entropy distribution across all tokens. We present the results in Figure 1 , from which we can observe: 1) Neither the overall average token entropy (Fig. 1 (a) ) nor the average entropy of high-entropy tokens (Fig. 1 (b)) can differentiate high- from low-quality samples. 2) Comparably, the summation of entropy over high-entropy tokens (Fig. 1 (d)) provides a more robust distinction compared to the total entropy of all tokens (Fig. 1 (c)). These findings demonstrate a strong correlation between sample quality and the cumulative entropy of critical tokens . Capitalizing on this relationship, we propose High-Entropy Sum (HES) , a train-free metric to quantify the long-CoT reasoning quality of the samples. Specifically, HES is calculated by summing the entropy values of the top k\% ( e.g. , 0.5%) of tokens with the highest entropy. Building on this metric, we propose an efficient, unified data selection method to curate high-quality reasoning data for LLMs, applicable across SFT, RFT, and RL training paradigms. To evaluate the effectiveness of our proposed HES-based method, we conduct extensive experiments across the three training paradigms. The results reveal that: 1) In SFT , training on the top 20% of samples ranked by HES achieves performance comparable to the full dataset across diverse domains, while expanding to the top 80% consistently surpasses the baseline. Notably, we demonstrate that data selection using a lightweight 0.6B model yields results comparable to those of an 8B model, validating the cost-effectiveness of HES for offline data curation; 2) In RFT , HES significantly outperforms traditional random selection, serving as an effective, training-free reward signal; 3) In RL , our strategy of oversampling rollouts and selecting only the top-50% ranked by HES outperforms the standard setting where all rollouts participate in policy updates. Collectively, these findings highlight that HES is not only high-performing but also remarkably efficient: it enables scalable proxy selection with small models in offline SFT and incurs negligible storage overhead in online RFT and RL settings. The empirical results confirm that our HES-based method is both an effective and efficient approach for selecting high-quality data, thereby advancing the reasoning capabilities of LLMs. In summary, our contributions are as follows: • We introduce High-Entropy Sum (HES), a train-free metric that quantifies reasoning quality by analyzing only the high-entropy tokens in the response paths. • We propose an effective and efficient HES-based data selection method that is compatible with SFT, RFT, and RL training paradigms. • Extensive experiments across diverse domains demonstrate that HES consistently outperforms baselines in SFT, RFT, and RL, achieving superior model performance with significantly reduced computational overhead. 2 Preliminaries Figure 2: Calculation of High-Entropy Sum (HES). The colored tokens are the tokens involved in the calculation. 2.1 Training Paradigms Supervised Fine-Tuning (SFT). Let \mathcal{D}=\{(x,y)\} denote a corpus of correct demonstrations, where x is a query and y is the reference response. SFT minimizes the cross-entropy loss, defined by the objective function: \mathcal{L}_{SFT}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}[-\log\pi_{\theta}(y|x)] , where \theta represents the model’s parameters and \pi_{\theta}(y|x) is the probability assigned by the model to y given x . Rejection Sampling Fine-Tuning (RFT). RFT ( 57 ) augments SFT by generating training samples through the model’s own exploration. The process involves three steps: (1) Generation: For query x , generate m different candidate responses \{y_{1},y_{2},\dots,y_{m}\} ; (2) Selection: Use selection function R(y) to select a response subset Y : Y=\underset{y_{i}\in\{y_{1},\dots,y_{m}\}}{\text{argmax}}\,R(y_{i}) ; (3) Fine-tuning: Create new dataset \mathcal{D}^{*} with Y and fine-tune using SFT. Reinforcement Learning (RL). Group Relative Policy Optimization (GRPO) ( 42 ) is a variant of Proximal Policy Optimization (PPO) (

Read the complete text