ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment

Xiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li, Longqiang Wang, Yue Wang, Ju Fan · 2026

Read ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment by Xiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li, Longqiang Wang, Yue Wang, Ju Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21484.txt.

Introduction

Xiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li, Longqiang Wang, Yue Wang, and Ju Fan align language models at test time without any retraining. Standard alignment training works but is costly and can be unstable. Their method instead samples directly from the policy that training would have produced.

An energy term guides each sampling step, combined with a reference model. The sampling runs inside a masked language model setup. Skipping training saves the full cost and risk of a training run. If the samples match trained quality, alignment becomes a setting teams can adjust on the fly rather than a project they must fund.

Catalog record

CREATORXiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li, Longqiang Wang, Yue Wang, Ju Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21484

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment Xiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li, Longqiang Wang, Yue Wang, Ju Fan Abstract Reinforcement Learning (RL) post-training alignment for language models is effective, but also costly and unstable in practice, owing to its complicated training process. To address this, we propose a training-free inference method to sample directly from the optimal RL policy. The transition probability applied to Masked Language Modeling (MLM) consists of a reference policy model and an energy term. Based on this, our algorithm, Energy-Guided Test-Time Scaling (ETS), estimates the key energy term via online Monte Carlo, with a provable convergence rate. Moreover, to ensure practical efficiency, ETS leverages modern acceleration frameworks alongside tailored importance sampling estimators, substantially reducing inference latency while provably preserving sampling quality. Experiments on MLM (including autoregressive models and diffusion language models) across reasoning, coding, and science benchmarks show that our ETS consistently improves generation quality, validating its effectiveness and design. The code is availale at https://github.com/sheriyuo/ETS . † † affiliationnotice: Equal contribution 1 Introduction Recently, Reinforcement Learning (RL) has become a central paradigm for aligning large language models (LLMs) with human intent ( 21 ; 45 ; 36 ) in a post-training stage ( 32 ) . Technically, in the RL stage, the LLM is recognized as a policy model ( 56 ) , which interacts with a reward model ( 48 ; 46 ; 50 ) . Despite their empirical success in reasoning ( 21 ) , image generation ( 57 ) , and natural language tasks such as summarization and dialog generation ( 25 ) , this paradigm suffers from many fundamental limitations: it requires costly reward modeling ( 45 ; 4 ) ; large-scale human preference data ( 46 ; 71 ) ; exhibits unstable training dynamics ( 72 ; 15 ) and sensitivity to hyperparameters ( 50 ) ; and must be rerun whenever the reward design changes ( 27 ) , leading to substantial computational and human costs. Theoretically, the existing KL-regularized RL objective of LLM admits a closed-form solution ( 50 ; 46 ) . However, existing training-based approaches ( 75 ) rely on iterative gradient-based optimization to approximate this optimal solution, rather than directly exploiting its structure. This raises a fundamental question: if the optimal RL distribution is known in closed form, can we sample from it directly at inference time, without any additional training? By doing so, all the aforementioned issues brought by post-training are resolved. Inspired by this, we explore inference-time sampling directly from the optimal RL distribution. Theoretically, we show that for the general masked language modeling (MLM) framework ( 3 ) , which includes autoregressive models (ARMs) ( 1 ; 66 ) and diffusion language models (DLMs) ( 43 ; 44 ) , the optimal transition kernel decomposes into two factors: (1) The original transition kernel provided by the reference model. (2) An energy term ( 38 ; 65 ) given by a conditional expectation of exponentiated rewards. This formulation enables training-free sampling from the optimal RL policy model. With the transition formulation, we design Energy-Guided Test-Time Scaling (ETS) to sample from the target optimal distribution without any training process. Moreover, we prove the convergence rate measured by total variation distance ( 16 ) of our method. Notably, our method is conceptually analogous to Monte Carlo Tree Search (MCTS) ( 67 ) and can be naturally interpreted as a form of Test-Time Scaling (TTS) ( 70 ) : additional computation at inference is used to explore multiple candidate continuations and guide the generation toward outcomes with high reward. Similar to other TTS methods, the central challenge of our method is how to perform this exploration efficiently without incurring prohibitive latency. Here, the bottleneck is the energy term in the transition kernel. To resolve this, we design efficient estimators for the energy term and corresponding sampling algorithms tailored to different model families. To accelerate decoding, for ARMs, we adopt a small proposal model ( 66 ) ; for DLMs, we use Fast-dLLM ( 64 ) with KV caching and parallel decoding. When combined with importance sampling ( 58 ) , both lightweight proposal models produce Monte Carlo unbiased energy estimators with substantially reduced latency. Theoretically, we prove these estimators yield samples converging to the target distribution. Empirically, we evaluate ETS across both ARMs and DLMs on reasoning, coding, and science benchmarks ( 23 ; 47 ; 9 ) . Our method consistently improves generation quality over standard inference and test-time scaling baselines ( 29 ; 30 ) , surpassing post-trained RL policy without any training. Comprehensive ablations validate our sampling procedures, reward design, and acceleration strategies, demonstrating favorable quality-latency trade-offs. 2 Related Work Reinforcement Learning (RL). RL was originally proposed as a framework for trial-and-error learning in sequential decision-making and optimal control ( 56 ; 41 ; 62 ; 48 ) . Recently, RL has been brought into the LLM post-training, most notably through RLHF and its variants, to align LLM behavior with human preferences and to improve reasoning capabilities ( 45 ; 21 ; 71 ; 54 ) . Common RL-based post-training algorithms for LLMs include PPO, DPO, and GRPO ( 48 ; 46 ; 21 ) . However, these methods suffer from high retraining costs, reliance on expensive human preference data, strong sensitivity to hyperparameters, and lack of well-defined rewards ( 7 ; 46 ) . Test-Time Scaling (TTS). TTS improves model performance by allocating additional computation at inference time rather than updating parameters ( 53 ) . Representative TTS techniques include Best-of-N ( 42 ) , Self-Consistency ( 60 ; 49 ) , Beam Search ( 55 ) , and Monte Carlo Tree Search (MCTS) ( 67 ; 8 ) . Beam Search maintains multiple candidate sequences during decoding and approximates maximum a posteriori generation via width-limited pruning, while MCTS enhances reasoning accuracy by sampling diverse solution paths and selecting the most consistent or highest-scoring outcome. Similar to this paper, Power Sampling ( 30 ) and Quest ( 17 ) also sample from RL post-training target distribution by Metropolis–Hastings (MH) algorithm ( 11 ) , leading to slow generation due to repeated recursive LLM queries. Moreover, ( 5 ; 18 ) propose to fit TTS with the target distribution of RL, but their gains heavily rely on training-based RL components rather than offering a general, training-free sampling solution. Recently, ( 14 ; 59 ; 52 ) proposed a similar framework to sample from the RLHF post-trained objective by intervening in the generation of trajectories. However, their methods are limited to diffusion models ( 24 ; 3 ) . Besides, unlike our method, their methods lack a theoretical convergence rate, and cannot be applied without a verifiable reward function. More importantly, our methods are more efficient than theirs due to our acceleration methods. Acceleration. The goal of inference-time acceleration is to reduce the latency while keeping the model’s output distribution essentially unchanged ( 74 ) . Techniques such as speculative decoding ( 34 ; 35 ; 26 ) , quantization ( 73 ) , pruning ( 39 ) , and sparse attention ( 37 ) are widely used in LLM deployment for this purpose. Since the energy term in our

Read the complete text