Reward-free Alignment for Conflicting Objectives
Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin · 2026
Read Reward-free Alignment for Conflicting Objectives by Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02495.txt.
Description
Peter Chen and 3 co-authors align language models to several conflicting goals without building reward models, in a 2026 arXiv paper. Real alignment problems mix objectives like helpfulness and safety that pull against each other. Weighted losses can fail to find updates improving all goals at once, and explicit reward models add complexity while distorting stated preferences.
Their system, RACO, learns directly from pairwise preference data. A clipped variant of conflict-averse gradient descent resolves clashes between objective gradients. Theory guarantees convergence to Pareto-critical points that respect user-specified weights, and clipping provably speeds convergence in the two-objective case. Heuristic refinements follow. On summarization and safety tasks across the Qwen, Llama, and Gemma model families, RACO achieved better Pareto trade-offs than current multi-objective rivals, in both measured and judged tests.
Catalog record
| CREATOR | Peter L. Chen, Xiaopeng Li, Xi Chen, Tianyi Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02495