FASTER: Value-Guided Sampling for Fast RL
Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn · 2026
Read FASTER: Value-Guided Sampling for Fast RL by Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19730.txt.
Introduction
Dong, Swerdlow, Sadigh, and Finn present FASTER, a way to gain the benefits of sampling-based test-time scaling for diffusion policies without paying the full compute cost. Top reinforcement learning methods sample many action candidates and pick the best. But that selection step is expensive at run time. FASTER traces the gain back to earlier in the denoising process, modeling candidate filtering as a Markov decision process and learning a policy and value function in denoising space.
Catalog record
| CREATOR | Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34337 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.19730
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FASTER: Value-Guided Sampling for Fast RL Perry Dong, Alexander Swerdlow, Dorsa Sadigh, Chelsea Finn Abstract Some of the most performant reinforcement learning algorithms today can be prohibitively expensive as they use test-time scaling methods such as sampling multiple action candidates and selecting the best one. In this work, we propose FASTER , a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost by tracing the performance gain of action samples back to earlier in the denoising process. Our key insight is that we can model the denoising of multiple action candidates and selecting the best one as a Markov Decision Process (MDP) where the goal is to progressively filter action candidates before denoising is complete. With this MDP, we can learn a policy and value function in the denoising space that predicts the downstream value of action candidates in the denoising process and filters them while maximizing returns. The result is a method that is lightweight and can be plugged into existing generative RL algorithms. Across challenging long-horizon manipulation tasks in online and batch-online RL, FASTER consistently improves the underlying policies and achieves the best overall performance among the compared methods. Applied to a pretrained VLA, FASTER achieves the same performance while substantially reducing training and inference compute requirements. Code: https://github.com/alexanderswerdlow/faster 1 Introduction Recent reinforcement learning (RL) methods have demonstrated strong performance using expressive policy backbones such as diffusion models which are widely used in domains such as image/video generation and robotics. However, some of the most performant of these algorithms are prohibitively expensive both during training and at test time. This is because many of these algorithms rely on sampling multiple candidate actions and selecting the best one, for example as determined by which one has the highest value. While effective, such approaches incur substantial computational costs. This is especially problematic for large models like modern Vision-Language-Action (VLA) models, which already operate near the limits of acceptable latency. A modest sample count of eight will multiply inference costs accordingly and can render these methods impractical in latency-sensitive or resource-constrained settings. In this work, we ask: can we recover the benefits of sampling-based test-time scaling without suffering its computational cost? Figure 1 : Left: Overview of FASTER . Instead of denoising all N candidates and selecting the best action post-hoc (best-of- N ), FASTER learns a denoise critic Q^{dn} that scores action samples during denoising, often directly on the initial noise itself. Right: Performance of FASTER compared with baseline averaged across tasks for \pi_{0.5} . The FLOPs are normalized on the x-axis against the amount of FLOPs needed for the baseline to converge. This high computational cost comes from policies with generative models such as diffusion or flow matching that need to denoise all candidates. Prior works have shown that test-time scaling methods for generative models are performant. Methods such as best-of- N sampling and self-consistency sample multiple candidates and select the one that maximizes a given metric. While these methods are effective at maximizing performance, they scale poorly and require full denoising over all action candidates. Distillation-based approaches amortize this cost by training the policy to directly reproduce high-value behaviors, but require training a separate policy which can be expensive. The core difficulty underlying all of these approaches is that, without observing the fully denoised samples, it is hard to know which ones are worth executing since there is no direct supervision linking a given noise sample to its downstream outcome, and the mapping from noise to action is only defined implicitly through the denoising process. In this work, our key insight is that we can model the denoising of multiple candidate samples and the selection of the best one as a Markov Decision Process (MDP) where the goal is to progressively filter candidates before denoising is complete. With this MDP, we can learn a denoise Q-function and policy with traditional temporal difference learning that decide which actions to keep and remove while maximizing the returns. To supervise the value function without requiring exhaustive rollouts, we use the value function over final actions as the reward of the action filtering MDP, enabling the model to propagate outcome information back to the denoising space without direct outcome labels for every candidate. Practically, we find that filtering candidates at the noise level—which is the cheapest instantiation computationally—achieves performance equivalent to fully denoising all action samples and selecting the highest value action. The computational cost of denoising—which is the dominant bottleneck at inference time—is thus reduced to that of a single rollout, regardless of the number of action candidates. The result is a lightweight, modular approach that decouples the benefits of broad action sample selection from the cost of full policy execution. Our main contribution is a framework for obtaining the performance gains of sampling and selecting the best of multiple actions of RL methods without the computational overhead of denoising all actions. Unlike approaches that operate on actions, our method operates directly on the denoising process to filter down candidates, making selection both cheap and modular. We evaluate our method on challenging manipulation benchmarks, including Robomimic and LIBERO, across algorithms and model capacities, demonstrating consistent gains over single-sample baselines and near-parity with best-of- N methods at a fraction of the computational overhead. 2 Related Work Test-time scaling and best-of- N inference. Significant work has shown test-time scaling methods for generative models to be highly performant. These methods take several forms, including iterative refinement, but the most common involve sampling multiple candidates and performing filtering or refinement based on a group of samples. Best-of- N sampling is the most standard of such methods, with extensive use in many domains including in autoregressive language models [ 1 , 2 , 3 ] , diffusion models [ 4 ] , and policy learning [ 5 , 6 , 7 , 8 ] . Diffusion policies and value-guided sampling. Diffusion models have emerged as a strong policy class for a wide variety of domains because they can represent rich, multimodal data distributions [ 5 , 9 , 10 , 6 , 11 , 12 , 13 , 14 ] . Several recent works study how to improve these policies with value functions or reduce their inference cost. Value-Guided Policy Steering (V-GPS) re-ranks actions from frozen robot policies with a learned value function [ 15 ] , RoboMonkey scales test-time sampling and verification for vision-language-action policies [ 16 ] , and EXPO couples an expressive base policy with a lightweight edit policy and selects value-maximizing actions online [ 5 ] . Other work amortizes or changes the policy itself: One-Step Diffusion Policy distills multi-step denoising into a faster actor [ 17 ] , while DSRL post-trains a behavior-cloned diffusion policy by running RL in its latent-noise space [ 18 ] . Compared to these approaches, our method learns a separate critic over which actions to filter during denoising , enabling candidate selection before actions are denoised. As such, our method can be directly applied to methods that can benefit from sampling multiple actions. Initial-noise and noise-space methods for diffusion models. A rapidly growing body of literature shows that the initial seed of a diffusion model has a significant effect