Streaming Looking Ahead with Token-level Self-reward
Hongming Zhang, Ruixin Hong, Dong Yu · 2025
Read Streaming Looking Ahead with Token-level Self-reward by Hongming Zhang, Ruixin Hong, Dong Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00029.txt.
Introduction
Zhang, Hong, and Yu teach a language model to grade its own words as it writes in this twenty twenty-five paper. Word-by-word tools can not see ahead, so they can miss the best line. Look-ahead hunts fix that, but they lean on a split grade model that costs much.
Inside, the core model gains a word-grade skill of its own, which cuts the split tool out. A stream-fit hunt then uses those grades to pick the next word with care. That hunt also runs in parts side by side, which keeps the stream fast. Tests on three broad sets show wins in most bouts with the plain greedy mode. With a fine-tune step added, the win rate climbs more.
Catalog record
| CREATOR | Hongming Zhang, Ruixin Hong, Dong Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00029
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Streaming Looking Ahead with Token-level Self-reward Hongming Zhang, Ruixin Hong, Dong Yu Abstract Autoregressive decoding algorithms that use only past information often cannot guarantee the best performance. Recently, people discovered that looking-ahead algorithms such as Monte Carlo Tree Search (MCTS) with external reward models (RMs) can significantly improve models’ output by allowing them to think ahead and leverage future outputs and associated rewards to guide the current generation. Such techniques can help the reinforcement fine-tuning phase by sampling better trajectories and the inference phase by selecting the better output. However, their high computational cost limits their applications, especially in streaming scenarios. To address this issue, we propose equipping the policy model with token-level self-reward modeling (TRM) capability to eliminate the need for external models and extra communication. We name the new architecture as Reward Transformer . In addition, we propose a streaming-looking-ahead ( SLA ) algorithm to further boost search efficiency with better parallelization. Experiments show that SLA achieves an overall win rate of 79.7% against the baseline greedy decoding algorithm on three general-domain datasets with a frozen policy model while maintaining streaming efficiency. If we combine SLA with reinforcement fine-tuning techniques such as DPO , SLA achieves an overall win rate of 89.4%. We release the experiment code at: https://github.com/CognitiveKernel/SLA . 1 Introduction Language models have demonstrated remarkable capabilities in broad natural language processing tasks, from text generation to question answering ( Yang 2019 ; Brown et al. 2020 ; Ouyang et al. 2022 ; Chowdhery et al. 2023 ) . However, despite their impressive performance, autoregressive decoding algorithms that only use past information lead to suboptimal results, particularly in unseen complex tasks ( Song et al. 2024 ) . To further enhance LLM’s performance, recent research proposes framing the generation process of large language models as a trajectory optimization problem, similar to reinforcement learning (RL), and optimizing it using advanced search algorithms like Monte Carlo Tree Search (MCTS) ( Metropolis & Ulam 1949 ; Silver et al. 2016 ) . MCTS allows the model to look ahead before making a move by incorporating an independent reward model (RM) to evaluate the quality of future generation trajectories. The advanced searching techniques are crucial for both the training and the inference phases. During the training phase, we could utilize MCTS to find an optimal solution and then upgrade the policy model through reinforcement fine-tuning (RFT) ( Xie et al. 2024 ; Zhang et al. 2024a ; Wang et al. 2024c ) . Given a fixed policy model during the inference phase, such searching algorithms can also help the model find a better trajectory than simply using greedy decoding ( Zhang et al. 2022 ; Liu et al. 2024b ) . Despite its advantages, the RM-based MCTS approach has notable efficiency limitations when applied to the LLM scenario, especially in streaming ones. In the LLM domain, the policy and reward models typically have billions, even trillions of parameters ( Donisch et al. 2024 ; Wang et al. 2024a ) , which costs not only a high computation burden but also a high communication burden since we need to distribute these models to separate machines. Moreover, the time complexity of these search algorithms is exponential towards the trajectory length. The overall complexity will become unacceptable if we directly apply search algorithms to the token level. To address the challenge, existing works mostly reduce the trajectory length by using more coarse-grained action granularity, such as defining each sentence, equation, or code block as an action ( Xu 2023 ; Dainese et al. 2024 ; Zhang et al. 2024b ; Brandfonbrener et al. 2024 ) . However, these heuristic approaches are limited to specific structured domains and do not apply to general-domain tasks. This paper aims to unleash the power of looking ahead algorithms by removing the external reward model. To achieve this goal, we propose to enhance the transformer architecture ( Waswani et al. 2017 ) with an additional channel to model rewards simultaneously. This design offers three advantages. First, since the architecture can produce reward estimation with near-zero additional computational cost for each token during exploration, it supports simultaneous token-level reward modeling (TRM), greatly expanding the method’s applicability to general-domain tasks. Second, integrating generation and self-evaluation into a single model significantly reduces the communication overhead of multi-model interactions. Third, distributing the reward modeling into each transformer block could significantly improve the reward modeling performance compared with widely used adapter-based methods. In the rest of the paper, we denote the proposed architecture as Reward Transformer . Figure 1: Overall framework of the proposed streaming looking ahead algorithm. We leverage the proposed Reward Transformer that combines policy modeling and token-level self-reward modeling to conduct the efficient looking-ahead search. At each step, the policy model will generate future tokens and the associated reward simultaneously and then aggregate back to select the better action. Building upon the trained model, we implement a streaming look-ahead ( SLA ) algorithm, with a demonstration in Figure 1 . At each step, the policy model looks ahead by simultaneously generating future tokens and the associated reward, which it then aggregates to select the better action. The whole process is streaming since generated future tokens can be directly reused for the next step of decision-making. Unlike previous MCTS works that require pre-defined steps ( Xu 2023 ; Dainese et al. 2024 ; Zhang et al. 2024b ; Brandfonbrener et al. 2024 ) , SLA supports arbitrary action granularity down to a single token to guarantee generalization capability. Evaluation on three general-domain datasets ( Zheng et al. 2023 ; Li et al. 2024 ; Dubois et al. 2024 ) demonstrates that SLA achieves a 79.7% win rate against the baseline greedy decoding with a frozen policy model. If the RFT is allowed, the combination of SLA and DPO could achieve an 89.4% win rate. To better monitor the TRM performance, we also propose a new evaluation metric AuTRC, with more details in subsection 5.3 . Empirical experiments show that the proposed architecture could significantly improve the TRM quality compared with existing methods. The paper is organized as follows. In section 2 , we provide the problem formulation and essential background. In section 3 , we introduce the importance and limitation of existing looking ahead algorithms in LLM applications. In section 4 , we introduce the details of SLA and Reward Transformer . In section 5 , we present empirical experiments to demonstrate the effectiveness of SLA and associated design choices. Lastly, section 6 concludes this paper. 2 Preliminaries In this section, we first introduce how the LM generation process can be formulated as a token-level Markov Decision Process (MDP) and then explain how existing sampling algorithms relate to it. 2.1 LLM Decoding as Token-level MDP The language model generation process takes a sequence of tokens as inputs and generates a sequence of tokens as outputs. Mainstream transformer-based language models generate the output tokens one by one until the stopping criteria (e.g., an end-of-sequence token or maximum sequence length) are met. The traditional MDP is usually formulated as a tuple \mathcal{M}=(\mathcal{S},\mathcal{A},F,R,\gamma) , where \mathcal{S} is the set of all possible states, \mathcal{A} is the set of actions, F is the transition function, R