Dual-Difficulty Curriculum Learning for Direct Preference Optimization
Mengyang Li, Haozhan Geng, Zhong Zhang, Shuang Liu · 2025
Read Dual-Difficulty Curriculum Learning for Direct Preference Optimization by Mengyang Li, Haozhan Geng, Zhong Zhang, Shuang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07856.txt.
Description
Teaching a language model from human preferences works better when easy lessons come before hard ones, but difficulty has more than one face. Mengyang Li and co-authors split it into two in this 2025 paper: how complex the prompt is, and how clearly the better answer stands apart from the worse. They build a grid from those two axes for direct preference optimization.
Fixed paths across the grid already beat standard training. Their main method goes further, letting the model pace itself and choose its own route as it improves. Tests show top scores on key benchmarks, better use of data, and more resistance to noisy preferences. The authors offer the two-axis view as a new frame for alignment work.
Catalog record
| CREATOR | Mengyang Li, Haozhan Geng, Zhong Zhang, Shuang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-07856