Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling

Runpeng Dai 1 Tong Zheng 2 Rui Liu 2 Chengsong Huang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 Washington University in St. Louis {runpeng, htzhu}@unc.edu Code: https://github.com/RunpengDai/RL-Guided-Adaptive-Sampling · 2026

Read Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling by Runpeng Dai 1 Tong Zheng 2 Rui Liu 2 Chengsong Huang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 Washington University in St. Louis {runpeng, htzhu}@unc.edu Code: https://github.com/RunpengDai/RL-Guided-Adaptive-Sampling (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03102.txt.

Introduction

Dai, Zheng, Liu, Huang, and Zhu cut the cost of test-time scaling for language models. Drawing more samples can raise answer quality, but it costs time and compute. Past adaptive methods stop sampling by fixed rules or by guesses about the spread of answers. This team frames the choice as a Markov decision process instead.

A small controller is trained with reinforcement learning. At each round it picks: stop, or draw more samples. It weighs answer gains against delay and cost. The controller needs only stats of the final answers, and it can train and run on a CPU. The team also shows the setup matches a relaxed form of a budget-bound optimization. Tests against strong baselines such as ASC and ESC show gains. The method found better trade-offs among accuracy, rounds of sampling, and total samples used.

Catalog record

CREATORRunpeng Dai 1 Tong Zheng 2 Rui Liu 2 Chengsong Huang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 Washington University in St. Louis {runpeng, htzhu}@unc.edu Code: https://github.com/RunpengDai/RL-Guided-Adaptive-Sampling
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling Runpeng Dai 1 Tong Zheng 2 Rui Liu 2 Chengsong Huang 3 Hongtu Zhu 1† 1 University of North Carolina at Chapel Hill 2 University of Maryland, College Park 3 Washington University in St. Louis {runpeng, htzhu}@unc.edu Code: https://github.com/RunpengDai/RL-Guided-Adaptive-Sampling Abstract Test-time scaling improves the reasoning performance of large language models but incurs substantial cost in both total computation and latency. Existing adaptive sampling methods partially mitigate this issue by dynamically deciding when to stop sampling, yet they typically rely on heuristic rules or rely on distribution assumptions. In this work, we formulate adaptive sampling as a Markov decision process (MDP). We train a lightweight sampling controller with reinforcement learning (RL) to jointly balance answer correctness, latency, and computation cost. At each round, the controller decides to stop sampling or to acquire additional samples. Our method is lightweight which only relies on statistics of final answers, and can be trained and deployed on CPU. We further show that the resulting framework admits an interpretation as the Lagrangian relaxation of a constrained optimization problem with explicit budget constraints. Experiments against strong baselines such as ASC and ESC show that our method achieves improved trade-offs among answer correctness, sampling rounds, and total samples required. 1 Introduction Test-time scaling ( 39 ; 56 ) has emerged as an effective way to improve the reasoning performance of large language models (LLMs) without additional training. Methods such as self-consistency ( 47 ) , tree-of-thoughts ( 53 ) , and Best-of- N sampling ( 33 ; 17 ) improve final answer quality by allocating more inference-time computation. However, these gains come with a clear downside: increased inference cost. As a result, effectively allocating inference-time computation to balance cost and performance is a critical challenge. A growing line of work seeks to reduce the cost of test-time scaling through adaptive sampling. Early methods, such as Adaptive Self-Consistency (ASC) ( 4 ) , utilize the posterior answer distribution to determine whether additional samples are needed. Subsequent works build on this by incorporating semantic signals ( 43 ) or altering the prior distribution ( 21 ) . Meanwhile, Early-Stopping Self-Consistency (ESC) ( 24 ) reduces latency by shifting to a parallel execution strategy. A concurrent line of research attempts to alter the reasoning process itself. At inference time, some methods trigger early stopping via signals such as confidence ( 13 ) , probing ( 30 ; 57 ) , or convergence dynamics ( 27 ; 55 ) . Alternatively, other works aim to increase reasoning efficiency during training time, leveraging either supervised fine-tuning ( 50 ; 32 ) or reinforcement learning ( 6 ) . Figure 1: Overview of the RL-Guided Sampling framework. Top two blocks illustrates mechanism of two adaptive sampling baselines. ASC sequentially samples one response at a time and stops when the posterior probability exceeds a predefined threshold. ESC samples in fixed batches and stops only when intra-batch consistency is achieved. Different from those approaches, RL-Guided Sampling guides the sampling process via a lightweight policy network. At each round, the framework constructs a state based on statistics from the observed answer pool. Given this state, the policy network directs the language model to either sample a specific number of additional responses in parallel or halt generation. As summarized in the bottom-left plot, RL-Guided Sampling outperforms baseline methods by requiring fewer total samples and sampling rounds." While existing approaches are effective, they often suffer from several key limitations: (i) Many rely heavily on human-designed heuristics or distributional assumptions, rather than explicitly deriving an optimal policy to navigate the performance-cost trade-off. (ii) Several methods require auxiliary signals that are often unavailable under specific scenarios, such as internal model confidence, hidden states, or question difficulty. (iii) Some techniques are highly invasive, disrupting the model’s natural reasoning process or even requiring additional training of the underlying LLM sampler. Such modifications incur significant operational overhead and are often incompatible with standard inference pipelines. These limitations motivate the need for a lightweight but principled method for efficient test-time scaling. Furthermore, to align with practical use cases, the method should flexibly account for multiple objectives, such as latency and computational cost. In this work, we present a framework that circumvents these limitations by formulating adaptive sampling as a Markov decision process (MDP). Rather than relying on predefined rules, we train a four-layer MLP controller via RL to learn a policy that optimizes the performance-cost trade-off. Crucially, our method relies purely on statistics derived from the sampled answer set, requiring only the final generated answers. This means it demands no auxiliary features, such as model confidence, and requires no intervention in the reasoning process of the LLMs. In the environment, we jointly consider multiple objectives. We use the correctness of the final answer as a positive reward, while treating inference costs (additional samples and sampling rounds) as penalties. Given an observed answer set, the controller dynamically decides whether to sample a specific number of additional responses or to stop sampling, as illustrated in Figure 1 . Practically, RL-Guided Sampling is extremely lightweight. The controller itself is only a small four-layer MLP, which can be trained and deployed efficiently on CPUs. Theoretically, our formulation intuitively maps to a constrained optimization problem: maximizing answer accuracy under strict latency and computation budgets. The weighted objective of our RL controller naturally arises as the Lagrangian relaxation of this problem, providing a principled mechanism to trade off performance and cost. We validate RL-Guided Sampling across three benchmarks and multiple language-model samplers. Our experiments show that the proposed controller consistently improves the accuracy–efficiency trade-off over strong adaptive sampling baselines. Specifically, as shown in Table 1 , RL-Guided Sampling reduces sampling rounds and total samples by 3 times and 30%, respectively, compared to ASC ( 4 ) , and by 10% and 35% compared to ESC ( 24 ) . This improvement is consistent across different trade-off levels (Figure 2 ). Furthermore, the learned policy generalizes well: controllers trained using samples from one dataset or model can be transferred to different benchmarks and even different samplers with minimal performance degradation (Section 4.4 ). 2 Method 2.1 Problem Setup and MDP Formulation We formulate the adaptive sampling problem as a finite-horizon Markov decision process (MDP). At each round, the controller observes the statistical features of current pool of sampled answers, and then decides whether to stop or to acquire some additional samples. The reward is designed to reflect three factors: the quality of the final aggregated answer, the latency cost induced by additional sampling rounds, and the computation cost induced by generating more candidates. Consider an input query x . Let N be the maximum number of samples allowed. At any given sampling round t , let \mathcal{D}_{t}=\{y_{1},\dots,y_{n_{t}}\} denote the observed answer set, where each y_{i}\in\mathcal{Y} is the final answer extracted from the i -th response, e.g., the content inside \boxed{} . Here, n_{t} denotes the total number of candidates collected up to round t . State. The state s_{t} summarizes the evidence currently available to the controller, together with the inference cost already incurred. Specifically, the state consists of three components: the counts of the most frequent answer classes, the

Read the complete text