Inference-Time Scaling for Generalist Reward Modeling

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu · 2025

Read Inference-Time Scaling for Generalist Reward Modeling by Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02495.txt.

Description

Zijun Liu and seven co-authors study in 2025 how reward models for language models can improve when given more compute at answer time. Reinforcement learning needs sound reward scores for open-ended queries, not just checkable ones. Generative reward models can flex across input types, but their scaling behavior was murky.

Their tuning method trains the model to state principles first, then critique answers against them. The resulting DeepSeek-GRM models gain from parallel sampling at run time. A small meta model guides voting among samples for better scaling. Tests show steady gains as compute grows. The work offers a path to reward scores that keep pace with ever larger policy models.

Catalog record

CREATORZijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02495

Browse the full library · About the Tower of Babel