Free Process Rewards without Process Labels
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng · 2024
Read Free Process Rewards without Process Labels by Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01981.txt.
Description
Lifan Yuan and co-authors published a computing paper in 2024 on getting step-by-step reward models for reasoning without step labels. Outcome models grade whole answers, while process models grade each step but need costly step labels. The team shows a process model hides inside an outcome model trained on plain answer labels. One assumption about how the outcome score is written makes the trick work.
Tests on math problems beat a strong tree-search baseline while using a tiny share of its training data. Scaling up prompts and answers helped, with answer count mattering more. Adding true step labels brought no further gain. The authors hope the work makes process models easier to train.
Catalog record
| CREATOR | Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01981