AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang · 2026
Read AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention by Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07815.txt.
Description
Hu and nine colleagues speed up long-context language-model inference with AsyncTLS, a two-level sparse attention system. Coarse block filtering first narrows the field, then fine token selection picks precisely, balancing accuracy against cost. An async offload engine overlaps transfers of stored key-value data with computation, using temporal locality. On Qwen3 and GLM-4.7-Flash, accuracy stays close to full attention. Operator speedups reach up to tenfold and end-to-end throughput gains up to nearly fivefold on contexts of 48,000 to 96,000 tokens.
Catalog record
| CREATOR | Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32697 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07815