Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi · 2026
Read Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph by Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08037.txt.
Description
Ning Liu, Chuanneng Sun, Kristina Klinkner, and Shervin Malmasi improve how language models learn from human preferences. The usual method, Direct Preference Optimization, trains on pairs of answers. In practice, one prompt often yields many sampled answers, and reducing them to pairs wastes ranking structure and can add conflicting signals.
Their method, GraphDPO, builds a directed graph from the full ranking of answers for each prompt. A graph-based objective follows dominance links between answers and recovers plain pairwise training as a special case. Ties and incomparable answers are handled directly, so training uses the whole preference structure instead of scattered pairs.
Catalog record
| CREATOR | Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08037