Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference footnoteinfo This work is partially supported by the FAIR project (NextGenerationEU, PNRR-PE-AI, M4C2, Investment 1.3), the 4DDS project (Italian Ministry of Enterprises and Made in Italy, grant F/310097/01-04/X56

Matteo Cercola, Valeria Capretti, Simone Formentin · 2025

Read Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference footnoteinfo This work is partially supported by the FAIR project (NextGenerationEU, PNRR-PE-AI, M4C2, Investment 1.3), the 4DDS project (Italian Ministry of Enterprises and Made in Italy, grant F/310097/01-04/X56 by Matteo Cercola, Valeria Capretti, Simone Formentin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04286.txt.

Description

Matteo Cercola, Valeria Capretti, and Simone Formentin propose a hybrid way to learn from human preferences that joins the scale of reinforcement learning from human feedback. It joins that scale with the query efficiency of preferential Bayesian optimization. Preference data are costly and slow to collect, yet such judgments steer models toward subjective human goals.

An acquisition-driven module inside the feedback pipeline chooses which comparisons to ask for, instead of gathering preferences passively. Tests cover high-dimensional preference tuning and language model fine-tuning. Both sample efficiency and overall performance improved steadily in its experiments. For alignment work, smarter questions can be worth as much as bigger answer sets.

Catalog record

CREATORMatteo Cercola, Valeria Capretti, Simone Formentin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04286

Browse the full library · About the Tower of Babel