Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance

Kai Yan, Alexander G. Schwing, Yu-Xiong Wang · 2026

Read Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance by Kai Yan, Alexander G. Schwing, Yu-Xiong Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15012.txt.

Introduction

Kai Yan, Alexander G. Schwing, and Yu-Xiong Wang propose FEST, a few-shot method for reinforcement learning with verifiable rewards. On hard problems, correct rollouts are rare, and demonstration-guided fixes usually demand large supervised datasets. FEST needs only 128 randomly chosen demonstrations, combining supervised cues, on-policy cues, and fading weights on the few-shot data to block overfitting. Across benchmarks it beats baselines using far less supervised data, even matching rivals trained on full datasets.

Catalog record

CREATORKai Yan, Alexander G. Schwing, Yu-Xiong Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71948
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance Kai Yan, Alexander G. Schwing, Yu-Xiong Wang Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has achieved great success in developing Large Language Models (LLMs) with chain-of-thought rollouts for many tasks such as math and coding. Nevertheless, RLVR struggles with sample efficiency on difficult problems where correct rollouts are hard to generate. Prior works propose to address this issue via demonstration-guided RLVR, i.e., to conduct Supervised FineTuning (SFT) when RL fails; however, SFT often requires a lot of data, which can be expensive to acquire. In this paper, we propose FEST, a FE w- S ho T demonstration-guided RLVR algorithm. It attains compelling results with only 128 128 demonstrations randomly selected from an SFT dataset. We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset to prevent overfitting from multiple-epoch training. On several benchmarks, FEST outperforms baselines with magnitudes less SFT data, even matching their performance with full dataset. 1 Introduction Two years after Reinforcement Learning from Human Feedback (RLHF) and GPT-3.5 [ 62 ] propelled Large Language Models (LLMs) to the forefront of AI research [ 41 ] , a new RL paradigm has emerged. Following OpenAI o1 [ 37 ] and DeepSeek-R1 [ 28 ] , Reinforcement Learning with Verifiable Rewards (RLVR) [ 28 ] has become the second dominant RL paradigm in the community. Unlike RLHF, which assigns rewards based on subjective and often vague human preferences [ 78 ] , RLVR leverages objective, verifiable rewards—such as unit tests for coding [ 40 ] or ground-truth comparisons for mathematics [ 18 ] . Consequently, RLVR is exceptionally well-suited for reasoning-heavy tasks. Driven by this approach, state-of-the-art LLMs have attained gold-medal performance in international competitions [ 34 ] and are beginning to tackle open problems at the frontiers of human knowledge [ 61 ] . Despite its impressive performance, RLVR is beset by a long-standing challenge in reinforcement learning: low sample efficiency on complex tasks. While prior knowledge from pre-training and Supervised Fine-Tuning (SFT) can partially mitigate this—analogous to imitation learning [ 4 ] —RL-trained LLMs still struggle to explore beyond the capabilities of their base models [ 97 ] . For instance, in mathematical tasks with binary rewards, a batch that fails to yield a single correct answer results in an advantage of 0 , providing no learning signal. To address this, state-of-the-art algorithms such as DAPO [ 95 ] and CISPO [ 11 ] employ repeated sampling until a success is found, which can triple the computational overhead on average [ 103 ] . Figure 1: Overview of our work. We introduce a few-shot demonstration-guided RLVR paradigm to address three primary challenges: the lack of on-demand data, limited expert input, and high overfitting risk. To mitigate, FEST incorporates three vital components— supervised learning , on-policy learning , and decaying weight —to effectively leverage few-shot data. Specifically, FEST jointly optimizes the model using GRPO on the answer-only data ( D_{I} ) and a semi-online DPO loss on the few-shot SFT data ( D_{E} ). The latter utilizes adaptive \beta coefficients to weight different types of rollouts, bridging the gap between expert demonstrations ( y^{+} ) and agent-generated samples ( y^{-} ). Table 1: Comparison of FEST with existing demonstration-guided RLVR baselines. SFT data requirements for HPT and SuperRL are estimated through replication (with SuperRL figures derived from training on OpenR1-220K); for ReLIFT, SASR, MIFO, and CHORD, we utilize the ratios reported in their respective papers. Notably, while other methods address competition-level problems, SASR focuses on simpler tasks. We omit DyME [ 42 ] due to its similarity to HPT and its focus on vision-based tasks rather than mathematical reasoning. RLVR Method SFT Data Used SFT Data Prepared Random SFT Data? SuperRL [ 51 ] Approx. 50K 220K \times LUFFY [ 89 ] 46K 46K \times SRFT [ 26 ] 46K 46K \times HPT [ 54 ] 10K 46K \times ReLIFT [ 55 ] 8.6K 46K \times MIFO [ 96 ] 6.4K 46K \times CHORD [ 101 ] 5K 5K ✓ SASR [ 14 ] 2K ∗ 4K ∗ \times FEST (Ours) 128 128 ✓ To address these challenges, recent research proposes a novel paradigm known as demonstration-guided RL, or unified post-training [ 89 ] . In this framework, SFT is integrated with RL, particularly when RL sampling fails to produce positive rollouts [ 54 , 55 ] . However, these approaches demand a large volume of SFT data, which can be prohibitively expensive [ 5 ] . For instance, curating just 2,500 questions for “Humanity’s Last Exam” [ 66 ] required 1,000 graduate-degree holders , even when providing only brief rationales—a level of effort that scales poorly for training data involving long reasoning traces. While bootstrapping and distilling from existing models is an alternative [ 19 ] , such practices raise significant concerns regarding legality [ 20 ] , proprietary API costs [ 36 ] , and potential model collapse [ 74 ] . In contrast, answer-only RL data are more accessible; large-scale mathematical Q&A pairs can be mined from online forums, whereas raw community answers typically necessitate extensive filtering or rewriting before serving as high-quality SFT demonstrations [ 102 , 57 , 3 ] . In this paper, we propose FEST, a FEw-ShoT demonstration-guided RLVR algorithm designed to thrive with as few as 128 randomly selected examples from an SFT dataset. To extract substantial performance gains from such a limited, uncurated dataset, we identify three critical components: (i) a supervised learning signal to provide expert guidance; (ii) an on-policy signal to mitigate exposure bias and serve as adversarial training [ 8 ] for enhanced robustness; and (iii) a decaying weight to prevent overfitting. We find that incorporating a semi-online Direct Preference Optimization (DPO) [ 67 , 15 , 77 ] loss—where demonstrations serve as positive examples and agent rollouts as negative ones—satisfies all three requirements. Furthermore, while our primary RL framework, Group Relative Policy Optimization (GRPO) [ 28 ] , operates on token-level log probabilities, standard DPO relies on sequence-level values. To resolve this mismatch, we introduce a variant of FEST that decomposes the DPO loss, replacing the online component with a GRPO loss featuring negative advantages, as supported by prior work [ 108 ] . Our framework is illustrated in Fig. 1 . Our contributions are summarized as follows: (i) We introduce a novel post-training paradigm, few-shot demonstration-guided RLVR, which significantly boosts RLVR performance with minimal SFT data (see Tab. 1 for a comparison with existing methods). (ii) We develop FEST by elucidating and integrating three vital components essential for this few-shot training regime. (iii) Theoretically, we extend the unified post-training framework of HPT [ 54 ] by incorporating DPO (Appendix B.3 ). (iv) We empirically validate our approach across multiple benchmarks, demonstrating that FEST consistently outperforms various strong baselines. 2 Preliminaries Reinforcement Learning with Verifiable Rewards (RLVR). RLVR is an emerging post-training framework that facilitates solving tasks with objective ground truths, such as mathematics and programming, by eliciting complex Chain-of-Thought (CoT) reasoning [ 83 ] . For a given prompt x representing the input task 1 1 1 Unless otherwise noted, the

Read the complete text