Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference footnoteinfo This work is partially supported by the FAIR project (NextGenerationEU, PNRR-PE-AI, M4C2, Investment 1.3), the 4DDS project (Italian Ministry of Enterprises and Made in Italy, grant F/310097/01-04/X56
Matteo Cercola, Valeria Capretti, Simone Formentin · 2025
Read Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference footnoteinfo This work is partially supported by the FAIR project (NextGenerationEU, PNRR-PE-AI, M4C2, Investment 1.3), the 4DDS project (Italian Ministry of Enterprises and Made in Italy, grant F/310097/01-04/X56 by Matteo Cercola, Valeria Capretti, Simone Formentin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04286.txt.
Description
Matteo Cercola, Valeria Capretti, and Simone Formentin propose a hybrid way to learn from human preferences that joins the scale of reinforcement learning from human feedback. It joins that scale with the query efficiency of preferential Bayesian optimization. Preference data are costly and slow to collect, yet such judgments steer models toward subjective human goals.
An acquisition-driven module inside the feedback pipeline chooses which comparisons to ask for, instead of gathering preferences passively. Tests cover high-dimensional preference tuning and language model fine-tuning. Both sample efficiency and overall performance improved steadily in its experiments. For alignment work, smarter questions can be worth as much as bigger answer sets.
Catalog record
| CREATOR | Matteo Cercola, Valeria Capretti, Simone Formentin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04286