Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?
Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai · 2026
Read Decomposing the Delta: What Do Models Actually Learn from Preference Pairs? by Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08723.txt.
Description
Lee and eight colleagues ask what preference data teaches reasoning models in methods like DPO and KTO. They separate generator level delta, the skill gap between models producing chosen and rejected traces, from sample level delta, judged quality gaps within a pair. Raising generator delta lifts out of domain reasoning, while filtering by sample delta enables more data efficient training. Their recipe: raise generator delta when building pairs, exploit sample delta when picking examples.
Catalog record
| CREATOR | Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36860 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08723