AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention

Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang · 2026

Read AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention by Yuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07815.txt.

Description

Hu and nine colleagues speed up long-context language-model inference with AsyncTLS, a two-level sparse attention system. Coarse block filtering first narrows the field, then fine token selection picks precisely, balancing accuracy against cost. An async offload engine overlaps transfers of stored key-value data with computation, using temporal locality. On Qwen3 and GLM-4.7-Flash, accuracy stays close to full attention. Operator speedups reach up to tenfold and end-to-end throughput gains up to nearly fivefold on contexts of 48,000 to 96,000 tokens.

Catalog record

CREATORYuxuan Hu, Jianchao Tan, Jiaqi Zhang, Wen Zan, Pingwei Sun, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai, Jing Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32697
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.07815

Browse the full library · About the Tower of Babel