Free Process Rewards without Process Labels

Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng · 2024

Read Free Process Rewards without Process Labels by Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01981.txt.

Description

Lifan Yuan and co-authors published a computing paper in 2024 on getting step-by-step reward models for reasoning without step labels. Outcome models grade whole answers, while process models grade each step but need costly step labels. The team shows a process model hides inside an outcome model trained on plain answer labels. One assumption about how the outcome score is written makes the trick work.

Tests on math problems beat a strong tree-search baseline while using a tiny share of its training data. Scaling up prompts and answers helped, with answer count mattering more. Adding true step labels brought no further gain. The authors hope the work makes process models easier to train.

Catalog record

CREATORLifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, Hao Peng
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01981

Browse the full library · About the Tower of Babel