LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu · 2026
Read LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning by Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01336.txt.
Description
Ji and co-authors compress long contexts across model families. Real tasks now push inputs past 100,000 tokens. Inference cost grows as context length grows with it. Compression can cut prefill costs while keeping accuracy. Training-free attention methods still fail on hard tasks. Code reasoning is one such demanding case. LongAttnComp fine-tunes a light cross-attention scoring layer. Token-level chunking and a budget-aware picker join it.
A two-stage tuning recipe shapes the compressor. Stage one builds a retrieval base from needle-style data. Stage two adds multi-hop and reasoning data. On InfiniteBench Code-Debug, it matches full-context accuracy. It transfers across four target models from three families. Gains also hold on LongBench v2 multi-document tasks.
Catalog record
| CREATOR | Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01336