Hierarchical Global Attention: Drop-In Exact-Token Routing for Pretrained Long-Context Transformers

Woernle Frank, Vladimir Fedosov, Artemiy Grinenko · 2026

Read Hierarchical Global Attention: Drop-In Exact-Token Routing for Pretrained Long-Context Transformers by Woernle Frank, Vladimir Fedosov, Artemiy Grinenko (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30709.txt.

Introduction

Woernle Frank, Vladimir Fedosov and Artemiy Grinenko present Hierarchical Global Attention, a drop-in replacement for attention in pretrained long-context language models. Attention over very long inputs is costly because every past token's keys and values must be stored and fetched. Their method changes only how tokens are retrieved: the pretrained weights stay untouched and nothing is retrained.

Retrieval works in two levels. Compact chunk summaries nominate promising regions; group-level routing refines the choice before exact attention runs on the selected tokens alone. The full history can live in RAM or fast disk storage, while only a small working set reaches the graphics card. On a thirty-billion-parameter Qwen model running on one consumer graphics card, the model handled a sixty-four-thousand-token context. Routed attention stayed within a few hundredths of a nat of dense attention while fetching about three percent of tokens.

Catalog record

CREATORWoernle Frank, Vladimir Fedosov, Artemiy Grinenko
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hierarchical Global Attention: Drop-In Exact-Token Routing for Pretrained Long-Context Transformers Woernle Frank, Vladimir Fedosov, Artemiy Grinenko Abstract Hierarchical Global Attention (HGA) is a drop-in replacement for dense causal attention in pretrained long-context transformers. HGA preserves the original checkpoint parameters: the pretrained W Q W_{Q} , W K W_{K} , W V W_{V} , and W O W_{O} projections remain unchanged, no calibration parameters are introduced, and no retraining is required. Applied to Qwen3-30B-A3B-Instruct-2507-FP8 on a single RTX 5090 (32GB), the patched model runs out of the box at a 64K-token context, where token-level K/V storage is not feasible on this hardware. Unlike previous sparse-attention methods, HGA performs hierarchical two-level routing. It first retrieves relevant chunks using compact RoPE-aware summaries and then refines the selection by routing only the most relevant groups before performing exact token-level attention. This hierarchical retrieval significantly reduces the number of fetched tokens while preserving exact attention over the retrieved token set, making RAM- and NVMe-backed storage practical. The full historical token K/V resides in host RAM or NVMe storage, while only a small routed working set is transferred to GPU memory during attention. Consequently, GPU memory consumption depends primarily on model weights and the routed working set rather than on the total context length. Across all tested context lengths (4K - 64K tokens), routed attention remains within approximately 0.01 0.01 – 0.02 0.02 nats of dense attention while the sparsity used is just about 3%. These results suggest that the approximation introduced by hierarchical routing is small, and that the remaining quality gap is likely dominated by long-context positional encoding rather than by the routing algorithm itself. 1 Introduction Applied to Qwen3-30B-A3B-Instruct-2507-FP8 on a single RTX 5090, the patched model runs at 32K-token context out of the box, where dense K/V storage is simply impossible on this hardware. On a 40M SmallLM, direct weight copy gives only a +0.018 nat loss gap versus dense attention at 8K tokens, and a Triton-fused implementation runs 2.72\times faster for training and 2.43\times faster for prefill at 12K tokens. A needle-in-a-haystack retrieval test on Qwen3-30B-A3B-Instruct-2507-FP8 at 64K-token context achieves 100% pass rate (3/3 depths) with group-level routing at 1.9% sparsity. The practical limit for long-context pretrained LLMs is often not only the O(n^{2}) attention computation, but also the dense K/V cache that must remain in accelerator memory. This is especially visible for quantized large models. In the Qwen3-30B-A3B-Instruct-2507-FP8 deployment case, the FP8 model weights already occupy almost the whole memory budget of a 32 GB GPU; storing every historical token key and value in VRAM leaves too little room for long contexts. HGA is designed as a systems-level patch for this situation. It does not retrain or replace the checkpoint’s attention projections. It keeps the original Q/K/V/O projections and attention norms and changes only which historical keys and values are fetched for each processing block. The GPU therefore holds the pretrained model, a compact chunk-summary table, the current chunk, a few always-hot chunks, and a routed working set. The rest of the historical token K/V lives in host RAM. The core observation is that consecutive tokens usually do not need completely independent long-range retrieval. A 64-token chunk often discusses one local idea, function, paragraph, or dialogue turn, and many tokens in that chunk benefit from the same distant chunks. HGA therefore first routes from the current chunk to previous chunks and groups. Only after this coarse retrieval step does it run ordinary softmax attention over exact token keys and values from the selected regions. In short, the long-range problem is shifted from dense token-to-token materialization to chunk-to-chunk retrieval plus exact token attention over a bounded fetched set. This design is particularly useful for pretrained models. The router summaries are built from the checkpoint’s own projected keys; they are sums or means with a RoPE-aware phase rule, not outputs of new learned linear layers. No calibration weights are introduced. The final attention output never uses synthetic summary values in the current method: summaries decide what to fetch, and the original token K/V computes the output. The implementation and benchmark scripts are available at https://github.com/vfedosov77/HierarchicalGlobalAttention . Our goal is not to redesign transformer attention, but to provide a drop-in hierarchical routing mechanism that can be applied directly to existing pretrained checkpoints. Rather than approximating the attention output itself, HGA approximates only the retrieval stage. Once relevant regions have been selected, the attention output is computed exactly using the original token-level keys and values. This design allows existing pretrained models to benefit from sparse long-context attention without modifying their learned representations. Our contributions are : • a drop-in exact-token attention replacement for pretrained GQA/MoE transformers that reuses original Q/K/V/O weights and norms and requires no calibration parameters; • a Qwen3-30B-A3B-Instruct-2507-FP8 integration in which the FP8 model runs with RAM-backed routed K/V on an RTX 5090 at 32K-token context, out of the box and without fine-tuning; • a chunk-shared routing scheme: nearby queries reuse a small fetched working set, e.g. about 1K routed middle tokens for a 64-token block in the current Qwen3 configuration, plus fixed sink/local/current chunks; • a two-level chunk–group routing policy combining deterministic attention sinks and recent local chunks with content-routed retrieval from the middle of the context; • repository benchmarks showing a +0.01828 nat copy-only loss gap at 8K tokens and 2.72\times training-step speedup at 12K tokens for a 40M SmallLM; • a tiered K/V storage abstraction for RAM-backed inference, in which full historical token K/V lives in host RAM while only a bounded working set occupies VRAM, decoupling GPU memory consumption from context length; • a needle-in-a-haystack evaluation on Qwen3-30B-A3B-Instruct-2507-FP8 showing 100% retrieval accuracy at 64K-token context without any fine-tuning. 2 Background Causal attention. For input X\in\mathbb{R}^{n\times d} , a standard attention head computes \mathrm{Attn}(Q,K,V)_{i}=\sum_{j\leq i}\mathrm{softmax}_{j}\!\left(\frac{q_{i}k_{j}^{\top}}{\sqrt{d_{h}}}\right)v_{j}. (1) The exact token-level receptive field grows with i , so the total number of query-key comparisons is O(n^{2}) and the K/V cache is proportional to all previous tokens. Grouped-query attention. Grouped-query attention (GQA) shares key/value heads across groups of query heads [ 1 ] . HGA preserves this layout. The source checkpoint’s W_{Q} , W_{K} , W_{V} , and W_{O} tensors keep their shapes and names. In the Qwen3 wrapper, the original projections and Q/K RMSNorm modules are kept by reference. RoPE. Rotary position embeddings (RoPE) rotate query and key dimensions by position-dependent angles [ 12 ] . A summary key is useful only if its phase is compatible with the query’s RoPE geometry. HGA therefore builds chunk and group summaries with a mixed rule described in Section 3. 3 Method 3.1 Partitioning The sequence is divided into chunks of fixed length C ; the current implementation uses C=64 . Each chunk is subdivided into groups of size g_{s} . The 40M benchmark uses g_{s}=16 , K_{c}=20 routed chunks and K_{g}=32 routed groups. The Qwen3-30B exact-token wrapper opens selected chunks directly with \mathit{topk\_chunks}=16 and keeps fixed first and recent windows. Closed chunks maintain three kinds of data: • a chunk key summary for the first routing stage; • group key summaries for the second routing stage where group routing is enabled; • the original token-level keys and values for exact attention after routing. Only the last item is used in the final

Read the complete text