CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang · 2026
Read CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs by Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05258.txt.
Description
Extending a language model's context window usually means retraining or fiddling with its position encoding. Haoran Li and 3 co-authors offer a lighter touch in this 2026 paper with CoPE, a modification of rotary positional embeddings, the standard scheme that encodes position by rotation. Two schools of fixes exist, one fighting out-of-distribution positions, the other protecting semantic attention patterns.
CoPE unifies both aims with one minimalist move: softly clipping the low-frequency components of the rotary encoding. Soft clipping removes the outlier positions that break generalization and sharpens semantic signals, while avoiding the spectral leakage that hard clipping causes. Experiments showed significant gains that held up to contexts of 256 thousand tokens, which the authors take as validation of their theory. They bill the method as a free lunch, near-zero cost for state-of-the-art length generalization, and release code, data, and models.
Catalog record
| CREATOR | Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05258