Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation

Asmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid, Basel Shbita · 2026

Read Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation by Asmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid, Basel Shbita (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18428.txt.

Introduction

Bhardwaj and co-authors learn decoding policies at test time for self-improving text generation. Greedy and fixed sampling rules are static and often ignore task style and structure. A light reinforcement policy adjusts sampling settings step by step with model weights frozen. On BookSum, arXiv, and WikiHow summaries, it beat greedy and static baselines by up to 88%. Composite rewards with length, coverage, and repetition terms proved key to stable gains.

Catalog record

CREATORAsmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid, Basel Shbita
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29423
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18428

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation Asmita Bhardwaj, Yuya Jeremy Ong, Eelaaf Zahid, Basel Shbita Abstract Decoding strategies largely determine the quality of Large Language Model (LLM) outputs, yet widely used heuristics such as greedy or fixed temperature/top- p p decoding are static and often task-agnostic, leading to suboptimal or inconsistent generation quality across domains that demand stylistic or structural flexibility. We introduce a reinforcement learning–based decoder sampler that treats decoding as sequential decision-making and learns a lightweight policy to adjust sampling parameters at test-time while keeping LLM weights frozen. We evaluated summarization datasets including BookSum, arXiv, and WikiHow using Granite-3.3-2B and Qwen-2.5-0.5B . Our policy sampler consistently outperforms greedy and static baselines, achieving relative gains of up to +88% (BookSum, Granite ) and +79% (WikiHow, Qwen ). Reward ablations show that overlap-only objectives underperform compared to composite rewards, while structured shaping terms (length, coverage, repetition, completeness) enable stable and sustained improvements. These findings highlight reinforcement learning as a practical mechanism for test-time adaptation in decoding, enabling domain-aware and user-controllable generation without retraining large models. 1 Introduction Large Language Models (LLMs) have become powerful generative systems, capable of producing contextually rich text and achieving impressive performance in a wide range of tasks ( 10 ; 13 ; 12 ; 2 ) . Text generation involves decoding, the process of selecting output tokens from the model’s predicted probability distribution. The decoding strategy provides heuristics for token selection and plays a decisive role in shaping the final generated output. Conventional decoding methods such as greedy sampling, top- k sampling ( 3 ) , or nucleus sampling ( 5 ) rely on fixed heuristics, while other methods such as beam search ( 8 ) , contrastive decoding ( 7 ) , or Mirostat ( 1 ) introduce controlled randomness on the same principle. Given a practically infinite search space for possible word sequences, these decoding strategies play a crucial role in the quality, diversity, and robustness of the generated text. However, these strategies do not adapt to the evolving context of generation or the demands of different tasks, limiting the degree of freedom of the model’s capability to generate text with more variations. Often such heuristics are limited by their lack of understanding of the control capabilities and effects of the text generation process, making it unreliable and ad-hoc. As a result, they often underperform when balancing trade-offs such as fluency, diversity, factual accuracy, and stylistic control. Moreover, improvements in decoding behavior typically require retraining or fine-tuning the LLM itself, which is computationally expensive and impractical at scale. From the perspective of recursive self-improvement, decoding offers a particularly attractive intervention point: it allows a model to monitor its own generation behavior, evaluate outcomes, and adjust future actions without modifying internal parameters. Such test-time adaptation enables improvement to occur within a single episode of generation, aligning with emerging views of self-improving systems that operate through feedback, control, and policy updates rather than continual retraining. Despite this potential, principled methods for adaptive decoding that are both lightweight and measurable remain underexplored. We propose a learnable reinforcement learning-based decoder sampler, in which an agent learns a policy that dynamically controls and adapts decoding parameters of the LLM at test-time while keeping the LLM weights fixed. In this context, we frame decoding as a sequential decision-making problem: the agent observes the evolving generation state (i.e., prompt, generated text, and model logits) and selects an action (i.e., setting the temperature hyperparameter) which influences the decoding behavior of the LLM. The reward signals are derived from task-specific metrics to guide the agent toward producing higher-quality output. This approach offers three main advantages. First, it decouples the sampling capabilities of the decoding sampling controls from the LLM training process, which avoids shifting the underlying learned distributions of the LLM. Second, it adapts in real time, allowing flexible control and near-real-time adaption across various tasks and domains. Third, by formulating decoding as a Markov Decision Process (MDP) ( 11 ) , it provides a guiding framework for defining states, actions, and rewards that generalize beyond a single task. Viewed through a recursive improvement lens, the proposed decoder forms a closed feedback loop: generation produces intermediate outputs, these outputs are evaluated via reward signals, and the resulting feedback informs subsequent decoding decisions within the same generation process. While we focus on lightweight policy learning rather than long-horizon model updates, this formulation captures a core mechanism of self-improving systems under fixed model capacity. In this work, we demonstrate the effectiveness of this method in the context of summarization, a task where decoding needs vary widely between various domains and text styles, as a demonstration of this capability. Experiments with Granite-3.3 ( 4 ) at 2B and Qwen-2.5 ( 14 ) at 0.5B demonstrate consistent improvements over static baselines such as greedy decoding. Ablation studies also reveal the contribution of different reward components and their combinations. Our key contributions are threefold: (i) we introduce a reinforcement learning decoder sampler that formalizes decoding as a test-time decision process with explicit state, action, and reward spaces, enabling adaptive decoding without modifying LLM weights; (ii) we perform initial evaluations across two models and multiple datasets demonstrating gains over static strategies; and (iii) we conduct ablation studies that highlight how different reward signals influence decoding behavior. 2 Related Work Numerous decoding strategies have been developed for large language models (LLMs), encompassing both deterministic methods (e.g., beam search ( 8 ) , contrastive decoding ( 7 ) ) and stochastic approaches (e.g., top- k , top- p , Mirostat ( 1 ) ). These methods primarily rely on heuristic or sampling-based mechanisms to balance fluency, diversity, and alignment with user intent. A comprehensive taxonomy and empirical evaluation of such decoding techniques is presented by 15 , highlighting both the strengths and limitations of conventional approaches in different tasks and model settings. However, these decoding strategies are typically hand-designed and task-agnostic, making it difficult to adapt generation behavior to specific downstream objectives or contextual preferences. To address these limitations, recent work has explored learnable and controllable decoding strategies, where decoding behavior is shaped by trainable components rather than fixed heuristics. 16 provide a comprehensive overview of controllable text generation using transformer-based models, highlighting methods that steer generation via prompt design, decoding constraints, or auxiliary models. 6 further categorize speculative and refinement-based decoding schemes that enhance generation quality or efficiency through multi-stage processes. These works underscore a common theme: most learnable decoding approaches operate by either post-hoc control of token selection or re-ranking candidate generations. In contrast, our method introduces a reinforcement-learned decoder sampler that directly learns a token selection policy over a frozen LLM, enabling decoding decisions to be optimized end-to-end with respect to downstream reward signals. This differs from prior work by tightly coupling decoding behavior with task-specific feedback, rather than relying on predefined constraints or sampling heuristics. Our method shares conceptual similarities with “Controlled Decoding”, which introduces a learned prefix scorer to guide decoding over a frozen base model 9 . However, while their scorer is trained offline to approximate a static reward function, our approach learns a decoding policy directly through reinforcement learning, enabling it to receive task-specific feedback at generation time. This allows

Read the complete text