Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively

Jiawei GuSun Yat-sen University [email protected], Shangsong LiangSun Yat-sen University [email protected] · 2025

Read Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively by Jiawei GuSun Yat-sen University [email protected], Shangsong LiangSun Yat-sen University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00396.txt.

Introduction

Jiawei Gu and Shangsong Liang propose a cheaper way for language models to make decisions, in a 2025 paper. Good decisions matter for hard tasks like math and planning. Many search methods buy small gains at great computing cost, though. The authors first set out a three-part test for judging that trade-off.

Their tool, the Speculative Reward Model, plugs into existing search methods. An outside reward giver predicts which action looks best, so the main model judges itself less. A checking step prunes weak choices early and steers search toward stronger steps. Tests covered math reasoning, planning, and number tasks in special fields. Costs fell to about a tenth on average, while results held steady.

Catalog record

CREATORJiawei GuSun Yat-sen University [email protected], Shangsong LiangSun Yat-sen University [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively Jiawei GuSun Yat-sen University [email protected], Shangsong LiangSun Yat-sen University [email protected] Abstract Effective decision-making in Large Language Models (LLMs) is essential for handling intricate tasks. However, existing approaches prioritize performance but often overlook the balance between effectiveness and computational cost. To address this, we first introduce the 3E Criteria to systematically assess the cost-effectiveness of search strategies, revealing that existing methods often trade significant efficiency for marginal performance gains. To improve LLM decision-making while maintaining efficiency, we propose the Speculative Reward Model (SRM), a plug-and-play framework that seamlessly integrates with existing search strategies. Specifically, SRM employs an external reward assigner to predict optimal actions, reducing reliance on LLMs’ internal self-evaluation. And a speculative verification mechanism is used to prune suboptimal choices and guide the search toward more promising steps. We evaluate SRM on several complex decision-making tasks including mathematical reasoning, planning and numerical reasoning in specialized domains. Experimental results show that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness. 🖂 🖂 footnotetext: Corresponding author. 1 Introduction Large Language Models (LLMs) ( OpenAI et al. 2023 ; OpenAI 2024 ; DeepSeek 2024 ; Qwen 2024 ) have achieved significant progress in natural language processing, excelling in text generation and comprehension ( Xu et al. 2025 ) . However, their application to complex reasoning and decision-making remains challenging ( Shao et al. 2024 ; Zelikman et al. 2024 ) , particularly when solving intricate problems that require structured logical inference rather than pattern-based predictions ( Valmeekam et al. 2023 ; Shao et al. 2024 ) . Table 1: Speculative Reward Models (SRM) , a plug-and-play framework designed to balance effectiveness and efficiency. In GSM8K tasks, all paradigms followed the same setting with GPT-3.5-turbo and 4-shot learning. The token cost is expressed in ‘[Prompt Tokens]/ [Completion Tokens]’. "Ext." denotes Extensibility. For Toolchain ∗ , which lacks direct execution capability, we estimate cost using identical prompts but exclude running time. Paradigm Effectiveness Efficiency Ext. Time Cost Token Cost Acc.[%] Avg.[sec.] Avg.[K] CoT ( Wei et al. 2022 ) 70.1 3.2 0.7/0.1 ✓ DFS ( Yao et al. 2023 ) 69.9 150 70.2/5.0 ✓ + SRM 70.5 34.7 18.6/0.8 ✓ BFS ( Yao et al. 2023 ) 72.3 180 85.5/7.1 ✓ + SRM 70.1 44 22.2/1.1 ✓ BS ( Wan et al. 2024 ) 71.4 66.4 225.4/4.4 ✓ + SRM 72.3 44 30.8/1.1 ✓ MCTS ( Hao et al. 2023 ) 74.7 122.6 105.2/2.5 ✓ + SRM 80.5 45.2 20.6/0.9 ✓ Toolchain* ( Zhuang et al. 2023 ) 78.9 – 40.8/1.9 \times To address these limitations, early studies introduced prompting strategies to enhance reasoning, such as Chain-of-Thought ( Wei et al. 2022 ) and AlphaZero-Like Tree-Search Method ( Wan et al. 2024 ) , which guide LLMs to generate intermediate reasoning steps to improving inference structure and accuracy. However, these methods rely solely on prompting without external validation or optimization Song et al. 2025 , limiting their reliability. Recent approaches employ tree-based search algorithms ( Besta et al. 2023 ; Ding et al. 2023 ; Putta et al. 2024 ; Wang et al. 2024 ) to explore broader reasoning paths and refine intermediate steps. By systematically evaluating multiple candidates in test time scaling Snell et al. 2024 , these methods enhance both the quality and diversity of reasoning, leading to more robust decision-making. Despite these improvements, they inevitably introduce substantial computational cost. In Table 1 , we utilize our proposed 3E Criteria — E ffectiveness , E fficiency , and E xtensibility to assess the cost incurred during LLM inference. Effectiveness represents the success rate, Efficiency denotes the time and token cost, and Extensibility is the adaptability to new tasks. The results reveal that existing methods offer limited performance gains at disproportionately high costs. For example, ToT ( Yao et al. 2023 ) , which employs Depth-First Search (DFS), Breadth-First Search (BFS), provides marginal performance improvements (0-3%), but incurs a 50-60 \times in time cost and a 100-120 \times escalation in inference complexity. Similarly, RAP ( Hao et al. 2023 ) leverages Monte Carlo Tree Search (MCTS), yielding a modest performance improvements of 4-5% at the expense of a 150-300 \times increase in inference cost. Additionally, Toolchain* ( Zhuang et al. 2023 ) and reasoning enhanced models like QwQ QwenTeam 2024 , constrained by task-specific heuristics, fails to reduce cost effectively and lacks extensibility. In this work, we seek to address: Research Question How to improve the reasoning ability of LLMs while maintaining a balance between effectiveness, efficiency, and extensibility? Inspired by studies Huang et al. 2023 emphasizing the need for external validation in decision-making, we propose Speculative Reward Models (SRM) , a plug-and-play framework designed to balance effectiveness and efficiency ( Jahan et al. 2016 ) . SRM introduces external rewards to mitigate ineffective decision-making in a speculative manner ( Xu et al. 2024 ; Chen et al. 2023 ; Xia et al. 2023 ) . It consists of two key components: (1) SRM, an independent reward model that assigns scores based on decision consistency and goal alignment. (2) Speculative Verification, a mechanism that ranks candidate steps by evaluating the consistency between internal rewards from LLMs and external rewards from SRM, enabling efficient pruning of suboptimal choices and guiding the search toward more promising states, thereby reducing computational cost. We first train SRM on datasets with weak process rewards and then fine-tune it to SRM + using strong search rewards. This allows us to provide potential success probabilities for specific steps as external reward signals to LLMs during the search phase. Extensive validation has demonstrated that our approach significantly lowers the cost to a fraction of the original search framework’s, without sacrificing effectiveness. In summary, our contributions are as follows: (1) Efficiency . The SRM framework we proposed dramatically increases efficiency with a notable reduction in cost, requiring only about 1/10 of the original search paradigms. (2) Effectiveness . There is no sacrifice of effectiveness for SRM; in fact, by integrating reward signals for process supervision, it achieves a up to a 10% performance improvement over CoT and approximately a 2% increase compared to using searching algorithms only. (3) Extensibility 1 1 1 Refers to whether the method requires retraining to adapt to new problems across different domains. SRM provides generalizable weak rewards and a universal framework for deriving strong rewards. Fine-tuning with strong rewards transforms SRM into SRM + , enabling domain-specific adaptation without full retraining. 2 Problem Formulation The decision-making process can be formulated as a Markov Decision Process (MDP) Puterman 1990 , where the state space \mathcal{S} represents all possible problem states with s\in\mathcal{S} , and the action space \mathcal{A} consists of actions a\in\mathcal{A} that transition the state toward a solution. The LLM acts as a generator \mathcal{G} , producing candidate actions \mathcal{G}(a|s,prompt_{1}) and determining state transitions \mathcal{G}(s^{\prime}|s,a,prompt_{2}) . A reward function \mathcal{R}(s,a) evaluates the effectiveness of actions in progressing toward the goal. Tree-based search paradigms in LLMs decompose complex problems into a sequence of manageable sub-problems, each represented as an action modifying the current state toward the final solution. The search tree \mathcal{T}=(\mathcal{S},\mathcal{A}) in Figure 1 represents the decision process, where nodes are states

Read the complete text