LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization
Guanzheng Chen Xin Li Michael Qizhe Shieh Lidong Bing · 2025
Read LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization by Guanzheng Chen Xin Li Michael Qizhe Shieh Lidong Bing (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13922.txt.
Description
Guanzheng Chen and co-authors teach short context language models to handle long texts by learning from their own short context skill, in work from twenty twenty-five. Models strong on short text often fade on long inputs. Human labels for long texts are impractical to gather at scale.
Their method, called LongPO, builds paired answers for the same prompt from long input and a squeezed short twin. Preference learning on those pairs moves short context skill into the long setting. A guard term keeps short context skill from slipping during training. Applied to a seven billion parameter model, context length grew fourfold with short skill intact. Long task scores beat standard fine tuning and rivaled far larger annotated models.
Catalog record
| CREATOR | Guanzheng Chen Xin Li Michael Qizhe Shieh Lidong Bing |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13922