Inference-Time Scaling for Generalist Reward Modeling
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu · 2025
Read Inference-Time Scaling for Generalist Reward Modeling by Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02495.txt.
Description
Zijun Liu and seven co-authors study in 2025 how reward models for language models can improve when given more compute at answer time. Reinforcement learning needs sound reward scores for open-ended queries, not just checkable ones. Generative reward models can flex across input types, but their scaling behavior was murky.
Their tuning method trains the model to state principles first, then critique answers against them. The resulting DeepSeek-GRM models gain from parallel sampling at run time. A small meta model guides voting among samples for better scaling. Tests show steady gains as compute grows. The work offers a path to reward scores that keep pace with ever larger policy models.
Catalog record
| CREATOR | Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02495