Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi · 2026
Read Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs by Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08763.txt.
Introduction
Zhiyuan Hu and nine co-authors tackle a flaw in how chat models learn by trial and reward. Trial learning can raise scores on hard tasks. But the model soon leans on a few paths it knows well. It then stops trying new paths. Its first try score rises, while its best of many tries score stalls.
The team adds a prize for rare paths. A path that few tries use gets a boost, if it is sound. That push keeps search wide for more time. A reader who trains chat models gets a way to guard fresh thought. It may help on tasks where the first known path is not the best path.
Catalog record
| CREATOR | Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08763
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs Zhiyuan Hu, Yucheng Wang, Yufei He, Jiaying Wu, Yilun Zhao, See-Kiong Ng, Cynthia Breazeal, Anh Tuan Luu, Hae Won Park, Bryan Hooi Abstract Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@ k k across large sampling budgets and increases the area under the pass@ k k curve (AUC@ K K ) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale. Code is in Software part under submission page. Code can be found here ( https://github.com/zhiyuanhubj/Uniqueness-Aware-RL ). 1 Introduction RL-based post-training is increasingly seen as a scaling paradigm for improving LLM reasoning, as reflected in a growing line of reasoning-oriented models (e.g., DeepSeek-R1 7 , GPT-5 15 , Qwen3-Thinking 24 , and Kimi-K2-thinking 20 ). However, as in classical reinforcement learning, it inherits a fundamental exploration–exploitation trade-off, which becomes particularly pronounced in complex reasoning tasks. LLMs training tend to prematurely converge to a small set of high-probability reasoning patterns that yield strong short-term rewards 4 ; 28 , leading to policy collapse and limited coverage of the solution space. As a result, insufficient exploration has emerged as a major bottleneck for scaling RL on LLMs. LLM reasoning produces long, multi-step rollouts. Encouraging randomness at the token level can increase surface variation, yet still yield highly similar reasoning modes and solution structures. As a result, token-level diversity is an imperfect proxy for exploration, and we instead target trajectory/strategy-level diversity. Despite recent progress in exploration-aware RL for LLMs, such as entropy bonuses 3 , low-probability regularization 10 , or pass@k-based objectives 2 , most methods encourage diversity indirectly through easy-to-measure signals like token entropy or embedding distance. These signals can increase variation in wording or sampling behavior, but they do not necessarily produce diverse solution strategies or broader coverage of the search space. For x^{2}-5x+6=0 , two rollouts may both use the quadratic formula but differ only in algebraic presentation. One shows intermediate steps like x=\frac{5\pm\sqrt{25-24}}{2} and simplifies step-by-step, the other simplifies immediately to x=\frac{5\pm 1}{2} . Token-level entropy (or embedding distance) can treat them as “diverse”, even though they share the same high-level strategy. By contrast, factoring (x-2)(x-3)=0 is a genuinely different solution path. This distinction matters in practice. Under pass@k , performance depends on maintaining multiple conceptually distinct strategies across k samples, not merely producing superficial token-level variation. As a result, RL training often improves pass@1 while silently eroding rollout-level diversity of solution strategies, leading to stagnant or even degraded pass@k, especially on hard reasoning tasks where users rely on multiple samples. In what follows, we sample K rollouts per problem during training and evaluate pass@k with k test-time samples (typically k\geq K ), reporting AUC@ k as the area under the pass@k curve. In this work, we take a different perspective. We argue that the right object to regularize is not tokens, but sets of rollouts (i.e., multiple sampled solution attempts) for a given instance, and that the notion of diversity is not surface semantics but strategy-level coverage. Concretely, we introduce a uniqueness-aware RL objective, which estimates each rollout’s strategy uniqueness relative to other candidates for the same problem, while separately verifying correctness with a problem-specific verifier. For each problem, we generate multiple rollouts and use a judge model (an LLM or a lightweight classifier) to cluster them by their high-level solution plan, while explicitly ignoring differences that are purely stylistic or local. We quantify a rollout’s strategy uniqueness using the size of its cluster, so rollouts in smaller clusters correspond to rarer strategies. We then integrate uniqueness and correctness into policy optimization by shaping the advantage. Correct rollouts that instantiate rare strategies receive amplified effective advantages, redundant correct rollouts are downweighted, and incorrect rollouts remain penalized. This “rewarding the rare” scheme incentivizes each rollout set to contain multiple correct and strategically distinct solutions, improving pass@ k without sacrificing pass@1. Empirically, we evaluate our method on Qwen2.5-7B 25 , Qwen3-8B 24 , and OLMo-3-7B 14 across diverse reasoning benchmarks, including mathematics (AIME 13 and HLE (Humanity’s Last Exam) 16 ), physics (OlympiadBench 8 ), and medicine (MedCaseReasoning 23 ).Across settings, our method enhances exploration and maintains strong performance as the sampling budget increases, up to k{=}256 , avoiding strategy collapse that limits baseline approaches. Further analyses show increased coverage of human-annotated solution strategies, validating that our gains reflect strategy-level exploration, not superficial variation. 2 Related Work Exploration collapse and token-level treatments. Recent work on RL for LLM reasoning has highlighted a pronounced form of exploration collapse, where continued training drives the policy toward a single “canonical” solution pattern per problem: entropy shrinks, pass@1 may increase, but the diversity of rollouts and gains in pass@k stagnate. A first line of approaches addresses this through entropy-based techniques, such as entropy bonuses and entropy-based scaling laws that predict and control target entropy over the course of training 5 ; 22 , or clipping schemes (e.g., Clip-Low/High) that explicitly avoid both near-greedy and overly random token-level distributions. Closer to our focus on rare behavior, low-probability regularization 5 and follow-up work like “Beyond the 80/20 Rule” 22 show that a small fraction of high-entropy, low-probability tokens carry disproportionate learning signal and are crucial for sustaining exploration under verifiable reward. These methods introduce regularizers that protect or amplify such tokens instead of letting RL suppress them completely. However, all of these techniques operate at the token or local distribution level. They do not distinguish whether two rollouts, built from different token trajectories, instantiate the same high-level solution idea or genuinely different strategies, and thus they cannot directly control diversity at the level of solution strategies within a problem. Diversity-aware objectives, pass@k training, and tradeoff between quality and diversity. A complementary line of work incorporates diversity more directly into the RL objective. Diversity-aware RL methods such as DARLING learn a semantic partitioning of answers and feed both quality and diversity scores into online RL, improving both utility and novelty across instruction following, creative writing, and competition math 12 . Pass@k-oriented objectives 26 ; 21 (including diversity-aware policy optimization and Potential@k-style training) view multiple rollouts per prompt as a set, emphasizing problems where pass@k is already high but pass@1 remains low, and using the gap between them to focus optimization on samples that still have untapped potential. At a more classical level, novelty search and quality–diversity algorithms reward solutions that are both high-performing and behaviorally novel, maintaining archives of