SwS: S elf-aware W eakness-driven Problem S ynthesis in Reinforcement Learning for LLM Reasoning
Xiao Liang 1 ∗ {}^{1\,*} , Zhong-Zhi Li 2 ∗ {}^{2\,*} , Yeyun Gong 3† , Yang Wang 3 , Hengyuan Zhang 4 , Yelong Shen 3 , Ying Nian Wu 1 , Weizhu Chen 3† 1 University of California, Los Angeles 2 School of Artificial Intelligence, Chinese Academy of Sciences 3 Microsoft 4 Tsinghua University · 2025
Read SwS: S elf-aware W eakness-driven Problem S ynthesis in Reinforcement Learning for LLM Reasoning by Xiao Liang 1 ∗ {}^{1\,*} , Zhong-Zhi Li 2 ∗ {}^{2\,*} , Yeyun Gong 3† , Yang Wang 3 , Hengyuan Zhang 4 , Yelong Shen 3 , Ying Nian Wu 1 , Weizhu Chen 3† 1 University of California, Los Angeles 2 School of Artificial Intelligence, Chinese Academy of Sciences 3 Microsoft 4 Tsinghua University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08989.txt.
Introduction
Xiao Liang and a team of co-authors present a 2025 paper on training large language models to reason better in math. Training with rewards for checkable answers works well, but it needs many good practice problems with exact answers. Human-written problems are scarce, and machine-made ones often ignore what a given model still gets wrong. The team proposes a framework called SwS that targets those gaps.
The method flags questions the model keeps failing during training and treats them as signs of weakness. It pulls out the core ideas behind those failures and builds new problems around them. Training then focuses on the model's weak spots without help from a stronger teacher model. Across eight reasoning benchmarks, scores rose by about ten points on the smaller model tested and about eight on the larger one.
Catalog record
| CREATOR | Xiao Liang 1 ∗ {}^{1\,*} , Zhong-Zhi Li 2 ∗ {}^{2\,*} , Yeyun Gong 3† , Yang Wang 3 , Hengyuan Zhang 4 , Yelong Shen 3 , Ying Nian Wu 1 , Weizhu Chen 3† 1 University of California, Los Angeles 2 School of Artificial Intelligence, Chinese Academy of Sciences 3 Microsoft 4 Tsinghua University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08989
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SwS: S elf-aware W eakness-driven Problem S ynthesis in Reinforcement Learning for LLM Reasoning Xiao Liang 1 ∗ {}^{1\,*} , Zhong-Zhi Li 2 ∗ {}^{2\,*} , Yeyun Gong 3† , Yang Wang 3 , Hengyuan Zhang 4 , Yelong Shen 3 , Ying Nian Wu 1 , Weizhu Chen 3† 1 University of California, Los Angeles 2 School of Artificial Intelligence, Chinese Academy of Sciences 3 Microsoft 4 Tsinghua University Abstract Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and limited-verification answers in existing distillation-oriented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model’s capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a S elf-aware W eakness-driven problem S ynthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses as questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure cases and synthesize new problems to strengthen the model’s weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization by empowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks. Code https://github.com/MasterVito/SwS Project https://MasterVito.SwS.github.io Figure 1 : 32B model performance across mainstream reasoning benchmarks and different domains. 1 Introduction "Give me six hours to chop down a tree and I will spend the first four sharpening the axe." —Abraham Lincoln Large-scale Reinforcement Learning with Verifiable Rewards (RLVR) has substantially advanced the reasoning capabilities of large language models (LLMs) [ 16 , 10 , 46 ] , where simple rule-based rewards can effectively induce complex reasoning skills. The success of RLVR for eliciting models’ reasoning capabilities heavily depends on a well-curated problem set with proper difficulty levels [ 63 , 28 , 55 ] , where each problem is paired with an precise and verifiable reference answer [ 14 , 31 , 63 , 10 ] . However, existing reasoning-focused datasets for RLVR suffer from three main issues: (1) High-quality, human-labeled mathematical problems are scarce, and collecting large-scale, well-annotated datasets with precise reference answers is cost-intensive. (2) Most reasoning-focused synthetic datasets are created for SFT distillation, where reference answers are rarely rigorously verified, making them suboptimal for RLVR, which relies heavily on the correctness of the final answer as the training signal. (3) Existing problem augmentation strategies typically involve rephrasing or generating variants of human-written questions [ 62 , 30 , 38 , 27 ] , or sampling concepts from existing datasets [ 15 , 45 , 20 , 73 ] , without explicitly considering the model’s reasoning capabilities. Consequently, the synthetic problems may be either too trivial or overly challenging, limiting their utility for model improvement in RL. More specifically, in RL, it is essential to align the difficulty of training tasks with the model’s current capabilities. When using group-level RL algorithms such as GPRO [ 40 ] , the advantage of each response is calculated based on its comparison with other responses in the same group. If all responses are either entirely correct or entirely incorrect, the token-level advantages within each rollout collapse to 0, leading to gradient vanishing and degraded training efficiency [ 28 , 63 ] , and potentially harming model performance [ 55 ] . Therefore, training on problems that the model has fully mastered or consistently fails to solve does not provide useful learning signals for improvement. However, a key advantage of the failure cases is that, unlike the overly simple questions with little opportunity for improvement, persistently failed problems reveal specific areas of weakness in the model and indicate directions for further enhancement. This raises the following research question: How can we effectively utilize these consistently failed cases to address the model’s reasoning deficiencies? Could they be systematically leveraged for data synthesis that targets the enhancement of the model’s weakest capabilities? To answer these questions, we propose a S elf-aware W eakness-driven Problem S ynthesis (SwS) framework, which leverages the model’s self-identified weaknesses in RL to generate synthetic problems for training augmentation. Specifically, we record problems that the model consistently struggles to solve or learns inefficiently through iterative sampling during a preliminary RL training phase. These failed problems, which reflect the model’s weakest areas, are grouped by categories, leveraged to extract common concepts, and to synthesize new problems with difficulty levels tailored to the model’s capabilities. To further improve weakness mitigation efficiency during training, the augmentation budget for each category is allocated based on the model’s relative performance across them. Compared with existing problem synthesis strategies for LLM reasoning [ 73 , 45 ] , our framework explicitly targets the model’s capabilities and self-identified weaknesses, enabling more focused and efficient improvement in RL training. To validate the effectiveness of SwS, we conducted experiments across model sizes ranging from 3B to 32B and comprehensively evaluated performance on eight popular mathematical reasoning benchmarks, showing that its weakness-driven augmentation strategy benefits models across all levels of reasoning capability. Notably, our models trained on the augmented problem set consistently surpass both the base models and those trained on the original dataset across all benchmarks, achieving a substantial average absolute improvement of 10.0% for the 7B model and 7.7% for the 32B model, even surpassing their counterparts trained on carefully curated human-labeled problem sets [ 14 , 6 ] . We also analyze the model’s performance on previously failed problems and find that, after training on the augmented problem set, it is able to solve up to 20.0% more problems it had consistently failed in its weak domain when trained only on the original dataset. To further demonstrate the robustness and adaptability of the proposed SwS pipeline, we extend it to explore the potential of Weak-to-Strong Generalization , Self-evolving , and Weakness-driven Selection settings, with detailed experimental results and analysis presented in Section 4 . Contributions . (i) We propose a Self-aware Weakness-driven Problem Synthesis (SwS) framework that utilizes the model’s self-identified weaknesses to generate synthetic problems for enhanced RLVR training, paving the way for utilizing high-quality and targeted synthetic data for RL training. (ii) We comprehensively evaluate the SwS framework across diverse model sizes on eight mainstream reasoning benchmarks, demonstrating its effectiveness and generalizability. (iii) We explore the potential of extending our SwS framework to Weak-to-Strong Generalization , Self-evolving , and Weakness-driven Selection settings, highlighting its adaptability through detailed analysis. Figure 2 : Illustration of the self-aware weakness identification during a preliminary RL training. 2 Method 2.1 Preliminary Group Relative Policy Optimization (GRPO) . GRPO [ 40 ] is an efficient optimization algorithm tailored for RL in LLMs, where the advantages for each token are computed in a group-relative manner without requiring an additional critic model to estimate token