DAST : Difficulty-Aware Self-Training on Large Language Models
Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, Kam-Fai Wong · 2025
Read DAST : Difficulty-Aware Self-Training on Large Language Models by Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, Kam-Fai Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09029.txt.
Introduction
Boyang Xue leads a team teaching language models to practice hardest on the problems they find hardest. Their paper appeared in two thousand twenty-five. Self-training lets models learn from their own answers. Standard schemes sample easy questions far more than tough ones. Hard problems thus get too little practice. Ceiling performance suffers exactly where it matters.
Their DAST framework first estimates difficulty by sampling. It then augments data around the challenging queries. A training stage mixes supervised fine-tuning with preference optimization. Math tasks supply the test bed. Gains hold across settings in their experiments. The lesson is simple: difficulty should steer self-training, not be averaged away.
Catalog record
| CREATOR | Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, Kam-Fai Wong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09029
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DAST : Difficulty-Aware Self-Training on Large Language Models Boyang Xue, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Hongling Xu, Fei Mi, Yasheng Wang, Lifeng Shang, Qun Liu, Kam-Fai Wong Abstract Present Large Language Models (LLM) self-training methods always under-sample on challenging queries, leading to inadequate learning on difficult problems which limits LLMs’ ability. Therefore, this work proposes a difficulty-aware self-training (DAST) framework that focuses on improving both the quantity and quality of self-generated responses on challenging queries during self-training. DAST is specified in three components: 1) sampling-based difficulty level estimation, 2) difficulty-aware data augmentation, and 3) the self-training algorithm using SFT and DPO respectively. Experiments on mathematical tasks demonstrate the effectiveness and generalization of DAST, highlighting the critical role of difficulty-aware strategies in advancing LLM self-training. Co-corresponding authors. 1 Introduction What doesn’t kill you makes you stronger. — Friedrich Wilhelm Nietzsche The lack of extensive, high-quality human-curated training data for Large Language Models (LLMs) constrains the potential upper bounds of their capacities, particularly on complex reasoning tasks ( Cobbe et al., 2021 ) . Recently, self-training techniques of LLMs have garnered increasing attention, which iteratively fine-tunes LLMs on their self-generated outputs, attaining sustained improvements and diminishing the reliance on human interventions ( Gulcehre et al., 2023 ; Singh et al., 2024 ; Huang et al., 2023 ; Zelikman et al., 2022 ) . To ensure the quality of LLMs’ self-generated training data, previous works employ rejection sampling ( Sordoni et al., 2023 ) to filter out low-quality or incorrect responses with external reward models ( Gulcehre et al., 2023 ) or ground-truth labels ( Singh et al., 2024 ) . This may lead to LLM over-sampling originally adept simple queries while under-sampling challenging queries ( Ding et al., 2024 ; Tong et al., 2024 ) . LLMs’ insufficient learning in challenging instances is primarily in two aspects during self-training. First, when fixing the sampling number, only a few even or no correct responses are acquired on challenging queries, which iteratively exacerbates the distribution imbalance of the training data and severely overfitting on simple questions (Left hand of Figure 1 (a)). Second, the lengths of sampled self-generated responses on difficult questions are not enough (Right hand of Figure 1 (a)). Given that challenging problems require more thinking steps ( Snell et al., 2024 ; Damani et al., 2024 ) , the quality of these responses tends to be lower. As a result, LLMs can not adequately learn from challenging tasks, thereby restricting their capacity improvements. Considering the above two issues, this work proposes a d ifficulty- a ware s elf- t raining (DAST) framework which focuses on increasing both the quantity and quality of self-generated responses on challenging queries during self-training: 1) DAST employs a sampling-based, model-specific method to estimate the difficulty level of each query. 2) Two data augmentation approaches are employed to balance the distribution and improve the response quality of training data given the difficulty levels. Specifically, we perform up-sampling on challenging questions to control the data proportion of different difficulty levels. We also employ a difficulty-matched few-shot prompting method to control the lengths of responses, encouraging LLMs to increase thinking steps on challenging questions. These two methods are combined incrementally. 3) We finally iteratively perform the above difficulty estimation and data augmentation steps in several rounds for LLM self-training using supervised fine-tuning (SFT) and direct preference optimization (DPO) ( Rafailov et al., 2023 ) respectively. Experiments are conducted on both the in-domain and out-of-domain tasks on various mathematical datasets. Results demonstrate that DAST significantly enhances LLMs’ math ability and generalizability over several baselines. Our contributions are as follows: 1) This work first comprehensively incorporates difficulty level into LLM self-training, demonstrating the significance of considering difficulty for future works. Our codes are available on https://github.com/AmourWaltz/DAST ; 2) We propose two data augmentation methods in DAST to improve both quantity and quality on challenging queries using the estimated difficulty level; 3) We conduct experiments and validate that DAST can enhance LLM’s math ability and generalizability using SFT and DPO respectively. 2 DAST Framework Figure 1: Changes of data proportion and response length distribution of samples in different difficulty levels during a three-round self-training process. The vanilla rejection sampling to construct training data (a) is widely employed in Singh et al. (2024) ; Gulcehre et al. (2023) ; Sordoni et al. (2023) ; Zelikman et al. (2022) . (b) and (c) are the proposed DAST aim to control data proportion and response lengths for challenging queries. Note that in iteration 0, the training data \mathcal{D}_{\mathrm{u}} is the original dataset \mathcal{D}_{\mathrm{o}} with ground-truth labels, while during iteration 1, 2, and 3, the training data is combined of self-generated data \mathcal{D}_{\mathrm{a}} and the original dataset \mathcal{D}_{\mathrm{o}} . All the difficulty levels are measured on the initial policy \mathcal{M}_{0} on the GSM8K test set and are fixed during self-training. 2.1 Difficulty Level Estimation We employ a sampling-based, model-specific method to estimate the difficulty level of each question to the model. Given the initial policy \mathcal{M}_{0} and the training set \mathcal{D}_{\mathrm{o}}={\left\{\boldsymbol{x}_{i},\boldsymbol{\hat{r}}_{i},\boldsymbol{\hat{y}}_{i}\right\}}_{i=1}^{N} , where \boldsymbol{x}_{i},\boldsymbol{r}_{i},\boldsymbol{y}_{i} represent the question, rationale, and the ground-truth answer respectively. Each rationale \boldsymbol{\hat{r}}_{i}=\left[\hat{r}_{i,1},\dots,\hat{r}_{i,l}\right] contains l reasoning steps where l varies in \boldsymbol{\hat{r}}_{i} . For each (\boldsymbol{x}_{i},\boldsymbol{\hat{r}}_{i},\boldsymbol{\hat{y}}_{i}) and a prompt set \mathcal{P} containing K different few-shot prompts, we employ each few-shot exemplar \boldsymbol{p}_{k}\in\mathcal{P} with the question \boldsymbol{x}_{i} for the policy \mathcal{M}_{0} to generate the k -th response \left({\boldsymbol{y}_{i}}^{(k)},{\boldsymbol{r}_{i}}^{(k)}\right)=\mathcal{M}_{0}\left(\boldsymbol{p}_{k},\boldsymbol{x}_{i}\right) using temperature sampling ( T=0.2,\mathrm{top}\penalty\ p=0.9 ). We obtain the response set \boldsymbol{Y}_{i}=\{{\boldsymbol{y}_{i}}^{(k)}\}_{k=1}^{K} and the label set \boldsymbol{Z}_{i}={\left\{{z_{i}}^{(k)}\right\}}_{k=1}^{K} by comparing each extracted answers in \boldsymbol{Y}_{i} with the ground-truth \boldsymbol{\hat{y}}_{i} to determine the correctness ( {z_{i}}^{(k)}\in\left\{0,1\right\} , 1 for True and 0 for False ). The difficulty level d_{i} is estimated as follows 1 1 1 In this study, the challenging queries refer to the queries estimated in difficulty levels of Middle, Hard, and Unsolved . Details and splits of four difficulty levels are in Table 3 . \displaystyle d_{i}=P\left(\boldsymbol{Y}_{i}|\boldsymbol{x}_{i}\right)=\frac{\sum_{k=1}^{K}\mathbb{I}\left({\boldsymbol{y}_{i}}^{(k)}={\boldsymbol{\hat{y}}}_{i}\right)}{K} (1) 2.2 Data Augmentation We augment \mathcal{D}_{\mathrm{o}} with the strategy \mathcal{A}(\cdot) for each query \boldsymbol{x}_{i} according to d_{i} by controlling the data proportion and response lengths on \mathcal{M} to obtain an augmented dataset \mathcal{D}_{\mathrm{a}} for self-training as follows. Data Proportion Control As in the left hand of Figure 1 (a), the construction of self-training data using rejection sampling may bias simple questions. Therefore, we set different sampling numbers K for different difficulty levels d_{i} of \boldsymbol{x}_{i} . More specifically, the sampling number K will multiply by a coefficient \beta determined by d_{i} as presented in Table 3 . For d_{i}\in\{M,H,U\} which indicates that \boldsymbol{x}_{i} is a challenging question, \beta is larger to increase the number of correct responses sampled from the policy \mathcal{M} . The sampled responses will be added into \mathcal{D}_{\mathrm{a}} . As illustrated in Figure 1 (b), we can dynamically control the proportion of samples in all difficulty levels and balance the distribution of the training data in each self-training iteration. Response Length Control As in the right hand of Figure 1 (a), the lengths of responses generated using the vanilla few-shot sampling method are in averaged length for all difficulty levels during self-training (iterations 1, 2, and 3) and relatively shorter than lengths of the ground-truth responses in \mathcal{D}_{\mathrm{o}} (iteration 0).