Reward-free Alignment for Conflicting Objectives

Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin · 2026

Read Reward-free Alignment for Conflicting Objectives by Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02495.txt.

Description

Peter Chen and 3 co-authors align language models to several conflicting goals without building reward models, in a 2026 arXiv paper. Real alignment problems mix objectives like helpfulness and safety that pull against each other. Weighted losses can fail to find updates improving all goals at once, and explicit reward models add complexity while distorting stated preferences.

Their system, RACO, learns directly from pairwise preference data. A clipped variant of conflict-averse gradient descent resolves clashes between objective gradients. Theory guarantees convergence to Pareto-critical points that respect user-specified weights, and clipping provably speeds convergence in the two-objective case. Heuristic refinements follow. On summarization and safety tasks across the Qwen, Llama, and Gemma model families, RACO achieved better Pareto trade-offs than current multi-objective rivals, in both measured and judged tests.

Catalog record

CREATORPeter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02495

Browse the full library · About the Tower of Babel