Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai · 2026

Read Decomposing the Delta: What Do Models Actually Learn from Preference Pairs? by Chia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08723.txt.

Description

Lee and eight colleagues ask what preference data teaches reasoning models in methods like DPO and KTO. They separate generator level delta, the skill gap between models producing chosen and rejected traces, from sample level delta, judged quality gaps within a pair. Raising generator delta lifts out of domain reasoning, while filtering by sample delta enables more data efficient training. Their recipe: raise generator delta when building pairs, exploit sample delta when picking examples.

Catalog record

CREATORChia-Hsuan Lee, Mingyang Zhou, Renkun Ni, Zelei Cheng, Sihui Dai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36860
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08723

Browse the full library · About the Tower of Babel