Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment
Youngjae Cho, Jongsuk Kim, Ji-Hoon Kim · 2026
Read Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment by Youngjae Cho, Jongsuk Kim, Ji-Hoon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04909.txt.
Description
Preference optimization trains language models from pairs of better and worse answers, and the fixed reference model it leans on can quietly go stale. Youngjae Cho and 2 co-authors are its authors. Methods such as Direct Preference Optimization teach language models from pairs of better and worse answers, holding updates in check against a fixed reference model. As training proceeds, the fixed reference drifts out of date, and noisy human labels get amplified.
Their method, GAPO, replaces the fixed reference with a moving anchor: a slightly perturbed copy of the current model that acts as a pessimistic baseline. The gap between the model and its anchor shows which training pairs are fragile, so those pairs count for less. Across noisy test settings, GAPO held up better than standard methods while matching or beating them on alignment and reasoning benchmarks. The paper was accepted at NeurIPS 2026.
Catalog record
| CREATOR | Youngjae Cho, Jongsuk Kim, Ji-Hoon Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04909