Learning Generative Selection for Best-of-N
Shubham Toshniwal, Aleksander Ficek , Siddhartha Jain , Wei Du, Vahid Noroozi, Sadegh Mahdavi, Somshubra Majumdar, Igor Gitman · 2026
Read Learning Generative Selection for Best-of-N by Shubham Toshniwal, Aleksander Ficek , Siddhartha Jain , Wei Du, Vahid Noroozi, Sadegh Mahdavi, Somshubra Majumdar, Igor Gitman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02143.txt.
Introduction
Shubham Toshniwal and colleagues teach small thought models to pick the best answer from many tries. Sampling answers can lift thought, but weak picking caps the gain, and strong picking has sat with large models. They build picking tasks from math and code sets, keeping cases with right and wrong tries. They train one point seven billion parameter models with reward based learning to back right picks. Their models beat prompt and vote rivals on math and code tests, at times matching larger models. Gains carry to stronger outputs.
Catalog record
| CREATOR | Shubham Toshniwal, Aleksander Ficek , Siddhartha Jain , Wei Du, Vahid Noroozi, Sadegh Mahdavi, Somshubra Majumdar, Igor Gitman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16914 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.02143
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Generative Selection for Best-of-N Shubham Toshniwal, Aleksander Ficek , Siddhartha Jain , Wei Du, Vahid Noroozi, Sadegh Mahdavi, Somshubra Majumdar, Igor Gitman Abstract Abstract: Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of- N N selection quality. Generative selection methods, such as GenSelect [ 20 ] , address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO [ 26 ] to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling. Figure 1 : Accuracy of Qwen3-1.7B generations when selected using different strategies on math and coding benchmarks; RL-trained GenSelect consistently outperforms prompting and majority voting. 1 Introduction Test-time scaling has emerged as a powerful approach for improving the reasoning performance of large language models (LLMs) [ 15 , 5 ] . A simple and widely used strategy is parallel sampling : generating multiple independent responses and selecting the best [ 2 , 19 ] . In practice, the effectiveness of parallel sampling is often limited by the quality of Best-of- N selection (i.e., choosing the best answer among N candidates), except for domains with automatic verifiers. Existing Best-of- N methods typically rely on shallow aggregation mechanisms, such as majority voting or self-consistency [ 23 ] , or on external reward models that score each candidate independently [ 4 , 8 ] . Recently, a new paradigm has emerged that uses reasoning models to perform aggregation over the N responses, posing this aggregation as a reasoning problem [ 31 , 16 ] . Among these, GenSelect [ 21 ] prompts a reasoning model to explicitly select the best candidate (given N candidate solutions, output the index of the selected candidate) and has demonstrated strong gains over prior methods. In this work, we ask whether strong GenSelect behavior can be learned by small reasoning models. We propose a targeted reinforcement learning framework for generative selection on math and coding tasks, where models are rewarded for selecting a correct solution among imperfect candidates. We construct candidate sets from problems on which the candidate generator achieves \leq 50\% pass rate, ensuring nontrivial selection difficulty, and determine correctness using automatic verifiers. Using on-policy reinforcement learning, we train 1.7B-parameter reasoning models that consistently outperform prompting and majority-voting baselines and approach the selection performance of much larger 8B-scale models ( \sim 5 \times larger). Notably, these gains generalize to selecting higher-quality outputs produced by stronger generation models, suggesting that learned selection transfers beyond the training-time candidate distribution. Our contributions are: • We introduce a reinforcement learning framework that trains small reasoning models to perform generative Best-of- N selection. • We construct math and coding selection tasks with automatically verified correctness and controllable selection difficulty. • We show that 1.7B selectors substantially improve Best-of- N performance and transfer to selecting candidates from stronger generators. GenSelect Prompt You will be given a problem followed by an enumerated list of {num_solutions} candidate solutions. Your task is to systematically analyze these solutions to identify the best approach. Problem: {problem} Solutions: {solutions} End your evaluation with exactly: Judgment [IDX] where IDX is the index 0-{max_idx} of the best solution. Figure 2 : The GenSelect prompt used for selection in prompt baselines and RL training. 2 Data Construction We construct GenSelect-style selection tasks for math and code using Qwen3-1.7B to generate candidate solutions and automatic verifiers (symbolic math checking or unit tests) to label correctness. We filter prompts to ensure they contain at least one correct solution while remaining nontrivial. 2.1 Math We construct math selection data from OpenMathReasoning [ 13 ] by selecting 37K problems on which the base model ( Qwen3-1.7B ) achieves a pass rate below 50% under a symbolic verifier based on Math-Verify [ 9 ] . For each problem, we sample 2–16 candidate solutions from Qwen3-1.7B and assemble them into GenSelect-style prompts ( Figure 2 ). We require at least one correct solution per prompt and cap the fraction of correct candidates at 50% to keep the selection task nontrivial. We construct up to four prompts per problem by resampling candidate solution sets using the above recipe; for training efficiency, we discard prompts exceeding 16K tokens. 2.2 Code For code, we use problems and test cases from OpenCodeReasoning (OCR) [ 1 ] . We generate candidate solutions with Qwen3-1.7B and execute them against the provided unit tests to obtain binary correctness labels. We then construct GenSelect prompts analogously to the math setting ( Figure 2 ), including up to 16 candidate solutions from Qwen3-1.7B and enforcing at least one correct solution with a 50% correctness cap. We do this for both domains to bias the dataset toward harder problems and balance candidate sets to contain both correct and incorrect solutions, making the selection task nontrivial. We discard prompts exceeding 16K tokens and responses over 12K tokens. 3 Experimental Setup 3.1 RL Training We train Qwen3-1.7B models for both math and code using the VeRL framework [ 17 ] on NVIDIA H100 GPUs. We use the DAPO algorithm [ 26 ] with on-policy selection generation. We use a batch size of 128, 16 rollouts for math and 8 rollouts for code, a learning rate of 1\times 10^{-6} , and the AdamW optimizer. For sampling, we use temperature 1.5, top- p 1.0, and a max output length of 16,384 tokens for both models. AIME24 AIME25 HMMT25 LCB v6 Generator: Qwen3-1.7B Pass@1 45.42 ± 5.33 37.92 ± 6.16 25.00 ± 7.13 33.45 ± 1.08 Pass@8 70.00 60.00 53.33 45.59 Majority@8 63.33 52.56 25.00 N/A GenSelect 1.7B Prompting 48.75 ± 6.89 38.33 ± 5.04 26.67 ± 5.63 34.72 ± 0.71 GenSelect 4B Prompting 61.25 ± 4.34 53.75 ± 4.15 34.17 ± 2.95 39.70 ± 0.81 GenSelect 8B Prompting 65.83 ± 2.36 55.42 ± 3.05 38.75 ± 3.05 41.24 ± 0.67 GenSelect 1.7B RL Math 65.00 ± 3.98 54.17 ± 2.36 36.25 ± 4.15 36.45 ± 0.86 GenSelect 1.7B RL Code 57.08 ± 3.75 44.58 ± 3.54 26.25 ± 5.18 36.84 ± 0.44 Generator: Qwen3-4B Pass@1 72.08 ± 3.96 61.67 ± 3.98 41.67 ± 7.13 51.98 ± 0.58 Pass@8 86.67 83.33 60.00 66.30 Majority@8 80.00 72.64 50.00 N/A GenSelect 1.7B Prompting 74.58 ± 3.96 63.75 ± 3.75 42.92 ± 4.52 52.01 ± 1.09 GenSelect 4B Prompting 80.00 ± 1.78 71.25 ± 2.48 50.00 ± 2.52 55.04 ± 0.88 GenSelect 8B Prompting 82.92 ± 2.14 72.50 ± 2.36 53.75 ± 2.78 56.11 ± 1.09 GenSelect 1.7B RL Math 80.83 ± 3.45 73.33 ± 3.09 55.42 ± 1.73 53.36 ± 0.81 GenSelect 1.7B RL Code 77.50 ± 2.95 67.08 ± 3.30 47.92 ± 2.48 52.67 ± 0.34 Generator: