CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs

Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang · 2026

Read CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs by Haoran Li, Sucheng Ren, Alan Yuille, Feng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05258.txt.

Description

Extending a language model's context window usually means retraining or fiddling with its position encoding. Haoran Li and 3 co-authors offer a lighter touch in this 2026 paper with CoPE, a modification of rotary positional embeddings, the standard scheme that encodes position by rotation. Two schools of fixes exist, one fighting out-of-distribution positions, the other protecting semantic attention patterns.

CoPE unifies both aims with one minimalist move: softly clipping the low-frequency components of the rotary encoding. Soft clipping removes the outlier positions that break generalization and sharpens semantic signals, while avoiding the spectral leakage that hard clipping causes. Experiments showed significant gains that held up to contexts of 256 thousand tokens, which the authors take as validation of their theory. They bill the method as a free lunch, near-zero cost for state-of-the-art length generalization, and release code, data, and models.

Catalog record

CREATORHaoran Li, Sucheng Ren, Alan Yuille, Feng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05258

Browse the full library · About the Tower of Babel