LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu · 2026

Read LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning by Mengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01336.txt.

Description

Ji and co-authors compress long contexts across model families. Real tasks now push inputs past 100,000 tokens. Inference cost grows as context length grows with it. Compression can cut prefill costs while keeping accuracy. Training-free attention methods still fail on hard tasks. Code reasoning is one such demanding case. LongAttnComp fine-tunes a light cross-attention scoring layer. Token-level chunking and a budget-aware picker join it.

A two-stage tuning recipe shapes the compressor. Stage one builds a retrieval base from needle-style data. Stage two adds multi-hop and reasoning data. On InfiniteBench Code-Debug, it matches full-context accuracy. It transfers across four target models from three families. Gains also hold on LongBench v2 multi-document tasks.

Catalog record

CREATORMengmeng Ji, Ravi Shanker Raju, Jonathan Lingjie Li, Chen Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01336

Browse the full library · About the Tower of Babel