Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment
Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Sekitoshi Kanai, Masanori Yamada, Kosuke Nishida, Kazutoshi Shinoda · 2026
Read Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment by Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Sekitoshi Kanai, Masanori Yamada, Kosuke Nishida, Kazutoshi Shinoda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04410.txt.
Description
Takahashi and colleagues propose a stable, consistent way to tune language models to likes. Most tuners assume a likes model, such as Bradley-Terry, which may miss true likes and break consistency. Direct density ratio tuning avoids likes models and reaches consistency, but its ratio often blows up, shaking training. The team's method instead uses a relative ratio between liked data and a mix of liked and unliked data. This relative ratio is capped and never blows up, so training stays stable with tighter bounds than the direct method.
Catalog record
| CREATOR | Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Sekitoshi Kanai, Masanori Yamada, Kosuke Nishida, Kazutoshi Shinoda |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54469 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04410