Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi · 2026

Read Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph by Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08037.txt.

Description

Ning Liu, Chuanneng Sun, Kristina Klinkner, and Shervin Malmasi improve how language models learn from human preferences. The usual method, Direct Preference Optimization, trains on pairs of answers. In practice, one prompt often yields many sampled answers, and reducing them to pairs wastes ranking structure and can add conflicting signals.

Their method, GraphDPO, builds a directed graph from the full ranking of answers for each prompt. A graph-based objective follows dominance links between answers and recovers plain pairwise training as a special case. Ties and incomparable answers are handled directly, so training uses the whole preference structure instead of scattered pairs.

Catalog record

CREATORNing Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08037

Browse the full library · About the Tower of Babel