Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models

Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif · 2025

Read Visualising Policy-Reward Interplay to Inform Zeroth-Order Preference Optimisation of Large Language Models by Alessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03460.txt.

Description

Fine-tuning a large language model the usual way, with gradients, eats memory and compute. Alessio Galatolo and co-authors study a lighter route in this 2025 paper: zeroth-order methods that judge updates by trying them, not by back-propagating errors. They first watch how a policy model and a reward model shift against each other during normal preference training.

Those patterns guide a new method, ZOPrO, which tweaks a classic sampling scheme to home in on good updates faster. On summarization, translation, and chat assistant tasks, rewards rose steadily and training speed matched gradient methods. It still trails the best systems, but it is the first zeroth-order method shown to work for preference tuning beyond simple sorting tasks.

Catalog record

CREATORAlessio Galatolo, Zhenbang Dai, Katie Winkle, Meriem Beloucif
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03460

Browse the full library · About the Tower of Babel