Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment

Youngjae Cho, Jongsuk Kim, Ji-Hoon Kim · 2026

Read Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment by Youngjae Cho, Jongsuk Kim, Ji-Hoon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04909.txt.

Description

Preference optimization trains language models from pairs of better and worse answers, and the fixed reference model it leans on can quietly go stale. Youngjae Cho and 2 co-authors are its authors. Methods such as Direct Preference Optimization teach language models from pairs of better and worse answers, holding updates in check against a fixed reference model. As training proceeds, the fixed reference drifts out of date, and noisy human labels get amplified.

Their method, GAPO, replaces the fixed reference with a moving anchor: a slightly perturbed copy of the current model that acts as a pessimistic baseline. The gap between the model and its anchor shows which training pairs are fragile, so those pairs count for less. Across noisy test settings, GAPO held up better than standard methods while matching or beating them on alignment and reasoning benchmarks. The paper was accepted at NeurIPS 2026.

Catalog record

CREATORYoungjae Cho, Jongsuk Kim, Ji-Hoon Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04909

Browse the full library · About the Tower of Babel