Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models

Jiayun Wu, Peixu Hou, Shan Qu, Peng Zhang, Ning Gu, Tun Lu, [0.3em] Fudan University · 2026

Read Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models by Jiayun Wu, Peixu Hou, Shan Qu, Peng Zhang, Ning Gu, Tun Lu, [0.3em] Fudan University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20212.txt.

Introduction

Wu and co-authors merge fast scalar judgments and slow reasoned judgments in one reward model. Reward models guide language model alignment, but the two main types split on speed and accuracy. Generative reward models reason step by step and cost a lot. Their F/S-RM trains a single model to do both styles. A dual confidence switch decides when the slow mode is worth it. Hybrid inference scored 84 on average across benchmarks. Token use fell by about a fifth versus full generative judging.

Catalog record

CREATORJiayun Wu, Peixu Hou, Shan Qu, Peng Zhang, Ning Gu, Tun Lu, [0.3em] Fudan University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47131
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models Jiayun Wu, Peixu Hou, Shan Qu, Peng Zhang, Ning Gu, Tun Lu, [0.3em] Fudan University Abstract Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM) , a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: first-token prediction as a scalar score (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. F/S-RM achieves a 1.2% relative performance improvement over state-of-the-art models while reducing token consumption by 20.8%. Code and data will be publicly available. 1 Introduction As Large Language Models (LLMs) continue to advance, reliably evaluating the increasingly complex and diverse text they generate has become a central challenge for defining model capability boundaries and achieving value alignment. Although human evaluation has long been regarded as the gold standard for capturing human preferences, its high cost and poor scalability significantly hinder rapid model iteration. To address these limitations, Reward Models (RMs) have emerged as efficient proxies for approximating human preference distributions. Figure 1: Structural and Performance Comparison of SRMs and GRMs. (a) Architectural designs and output formats—SRMs produce scalar scores while GRMs generate reasoning chains. (b) Performance across three difficulty levels on RM-Bench. GRMs excel on Hard cases while SRMs dominate Easy cases, revealing complementary strengths. Traditional RMs predominantly adopt the form of Scalar Reward Models (SRMs). In this paradigm, the model acts as a discriminator: a pretrained language model is augmented with a linear head that maps input text pairs to a scalar score. While SRMs offer fast inference, they collapse a complex evaluation procedure into a single-token prediction over the reward score, failing to fully exploit the generative and reasoning capabilities of LLMs. Consequently, SRMs often exhibit suboptimal accuracy, especially on out-of-distribution data 22 . To address these constraints, recent research has increasingly shifted toward Generative Reward Models (GRMs) 19 , typically categorized under the LLM-as-a-Judge paradigm 13 . Diverging from the single scalar output of SRMs, GRMs recast the task as a generative evaluation process. By incorporating chain-of-thought (CoT) reasoning and reinforcement learning, GRMs learn to produce explicit reasoning traces before issuing final judgments, thereby mimicking human annotators’ deliberate cognitive processes 26 ; 30 ; 3 . This human-like "reason-then-judge" process has proven highly effective in improving evaluation accuracy, demonstrating the potential to match human expert performance 4 ; 25 . Despite these advantages, GRMs are not a comprehensive replacement for SRMs. Generating hundreds of reasoning tokens entails a computational cost orders of magnitude higher than the single-token calculation of SRMs, thereby limiting scalability for online use. Traditional SRMs, while benefiting from efficient inference, face their own challenges: they require massive amounts of high-quality supervision 16 ; 2 and often exhibit limited robustness in cross-domain generalization. As illustrated in Figure 1 b, we analyze the performance of current open-source SRMs and GRMs across different difficulty levels on RM-Bench. While GRMs demonstrate significant advantages on complex cases, they underperform SRMs on simple tasks. This complementarity mirrors the Dual Process Theory of human cognition 6 ; 5 : SRMs provide efficient pattern-based judgments (System 1), while GRMs enable deep analytical reasoning (System 2). However, existing systems treat them as mutually exclusive alternatives rather than synergistic components. This gap motivates our central research question: Can we efficiently integrate SRMs and GRMs? In this work, we propose Fast-Slow Thinking Reward Models (F/S-RM) , a hybrid RM architecture that integrates fast scalar judgments and slow chain-of-thought reasoning within a single base model. The model’s first output token serves as a fast scalar reward, then employs a dual-confidence activation mechanism to determine whether to invoke slow generative reasoning with CoT for refined decisions when internal confidence is insufficient. Experimental results demonstrate that F/S-RM achieves superior efficiency-accuracy trade-offs. F/S-RM maintains state-of-the-art performance with a 1.2% relative improvement while cutting token consumption by 20.8%. Our main contributions are: • We propose F/S-RM, a hybrid RM that unifies scalar and generative reward modeling paradigms within a model. • We design a dual-confidence activation mechanism that can balances efficiency and accuracy. • Extensive experiments demonstrate F/S-RM’s effectiveness across diverse benchmarks (RewardBench, RM-Bench, and JudgeBench), with thorough ablation studies validating each design. 2 Preliminary Reward models (RMs) play a central role in alignment pipelines such as Reinforcement Learning from Human Feedback (RLHF), converting human or AI preference signals into scalar rewards for optimization 20 . A RM serves as a proxy for human preference, functioning as a text classifier that evaluates the quality of a response given a specific prompt. Problem Formulation. Formally, given an input prompt x and a response y , the reward signal is defined as r=R_{\psi}(x,y) , where higher values indicate better alignment with human preferences. RMs are typically trained on preference datasets \mathcal{D}=\{(x,y_{w},y_{l})\} , where y_{w} and y_{l} denote the preferred and rejected responses for prompt x , respectively. Bradley-Terry Model. Preference learning is commonly modeled using the Bradley–Terry (BT) framework 1 , which assumes that observed preferences are induced by an underlying latent reward function. Under this model, the probability that y_{w} is preferred over y_{l} is given by p_{\mathrm{BT}}(y_{w}\succ y_{l}\mid x)=\sigma\big(r(x,y_{w})-r(x,y_{l})\big), (1) where \sigma(\cdot) denotes the sigmoid function. The reward model is trained via maximum likelihood by minimizing \begin{split}\mathcal{L}_{\mathrm{BT}}=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\Big[\log\sigma\big(R_{\psi}(x,y_{w})\\ \quad-R_{\psi}(x,y_{l})\big)\Big].\end{split} (2) 3 Fast-Slow Thinking RM Figure 2: Architecture and training workflow of F/S-RM. (a) Adaptive reasoning task. (b) Reward signal generation modeled as an adaptive reasoning chain. (c) Two-stage training pipeline for optimizing fast-thinking judgment and slow-thinking CoT reasoning. The reward signal generation process of F/S-RM can be viewed as an adaptive reasoning chain generation process. As illustrated in Figure 2 , the model is required to make a preference judgment at the first token and activate slow thinking when its internal confidence is insufficient. However, directly training on such severely length-imbalanced samples is prone to action space collapse and reward hacking. To address this, we design a two-stage progressive training framework: (i) Fast Thinking as First-Token Prediction : We model the first-token prediction as a SRM, training the model’s rapid judgment capability through Bradley-Terry preference loss and token action space constraint—to obtain the fast-thinking model \pi_{\theta}^{\text{fast}} ; (ii) RL for Slow Thinking : Building upon \pi_{\theta}^{\text{fast}} , we append a trigger token “tie” to the original prompt, enabling the model to bypass fast judgment and enter slow-thinking mode with chain-of-thought reasoning. We then optimize it via online reinforcement learning (GRPO) to obtain the slow-thinking model \pi_{\theta}^{\text{slow}} ; (iii) Dual-Confidence Activation Mechanism : After endowing the model with both reasoning capabilities on the same foundation, we employ a dual-confidence activation strategy to determine whether to trigger slow thinking following fast judgment, yielding the final adaptive model \pi_{\theta}^{*} . 3.1 Fast Thinking as First-Token Prediction Following 30 , which interprets token generation probabilities as scalar rewards, we formulate fast thinking as a first-token prediction task. This design requires only a single

Read the complete text