Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Sekitoshi Kanai, Masanori Yamada, Kosuke Nishida, Kazutoshi Shinoda · 2026

Read Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment by Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Sekitoshi Kanai, Masanori Yamada, Kosuke Nishida, Kazutoshi Shinoda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04410.txt.

Description

Takahashi and colleagues propose a stable, consistent way to tune language models to likes. Most tuners assume a likes model, such as Bradley-Terry, which may miss true likes and break consistency. Direct density ratio tuning avoids likes models and reaches consistency, but its ratio often blows up, shaking training. The team's method instead uses a relative ratio between liked data and a mix of liked and unliked data. This relative ratio is capped and never blows up, so training stays stable with tighter bounds than the direct method.

Catalog record

CREATORHiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Sekitoshi Kanai, Masanori Yamada, Kosuke Nishida, Kazutoshi Shinoda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54469
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04410

Browse the full library · About the Tower of Babel