Move the Query, Not the Cache: Characterizing Cross-Instance Latent and Sparse Attention Redistribution Across GPU Fabrics Journal: TACO CCS: Networks Network performance modeling CCS: Computer systems organization Distributed architectures CCS: Computing methodologies Machine learning CCS: Networks Data center networks

Bole Ma, Jan Eitzinger, Harald Köstler, Gerhard Wellein · 2026

Read Move the Query, Not the Cache: Characterizing Cross-Instance Latent and Sparse Attention Redistribution Across GPU Fabrics Journal: TACO CCS: Networks Network performance modeling CCS: Computer systems organization Distributed architectures CCS: Computing methodologies Machine learning CCS: Networks Data center networks by Bole Ma, Jan Eitzinger, Harald Köstler, Gerhard Wellein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01502.txt.

Introduction

Ma, Eitzinger, Köstler, and Wellein ask whether to move queries or caches. Sparse attention indexers pick a few KV-cache blocks per query. Agentic workloads reuse those blocks across many sub-agents. When a corpus outgrows one GPU, blocks sit on other instances. Prior systems move the cache to the requester. Multi-head Latent Attention inverts that arithmetic, though. Each token compresses into one narrow vector. A routed query row is about 1 KB, smaller than its chunk.

Routing the query is then often cheaper than moving cache. They characterize both options on a real H100 cluster. A topology-aware cost model distills the trade-offs. A closed-form predicate picks route, fetch, or local paths. The model tracks batched round trips within about 7 percent.

Catalog record

CREATORBole Ma, Jan Eitzinger, Harald Köstler, Gerhard Wellein
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01502

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Move the Query, Not the Cache: Characterizing Cross-Instance Latent and Sparse Attention Redistribution Across GPU Fabrics Journal: TACO CCS: Networks Network performance modeling CCS: Computer systems organization Distributed architectures CCS: Computing methodologies Machine learning CCS: Networks Data center networks Bole Ma, Jan Eitzinger, Harald Köstler, Gerhard Wellein Abstract . Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention’s unit is now a small, reusable chunk. Agentic workloads hammer it: many sub-agents query one large codebase, reusing the same blocks. When that corpus outgrows one GPU it is partitioned across instances, so a query and the blocks it selects often sit on different GPUs: answering it means attention across instances. The reflex of prior cross-instance KV systems is to move the cache : pull the selected blocks to the requester. Multi-head Latent Attention (MLA) inverts the arithmetic, compressing each token’s key and value into one narrow vector, so a routed query row is only ≈ \approx 1 KB, smaller than the chunk it attends; routing the query is then often cheaper than moving the cache. Which primitive wins, over which fabric and request shape, is uncharted, least of all on device-initiated RDMA (IBGDA) that makes per-request cross-node transfers cheap. We characterize cross-instance MLA attention on a real multi-node H100 cluster, distilling two reusable artifacts: a topology-aware cost model (probe / transfer / compute / return / merge) and a closed-form route / fetch / local predicate, whose constants we measure on real IBGDA, where the model tracks batched round-trips to within ∼ \sim 7%. At decode it routes the query, trading the cost of moving the cache (a ≈ \approx 3 ms re-adaptation splice for a contiguous chunk, or a scattered gather under selection) for a tens-of-microsecond round trip, and picks the fabric by probe latency, not peak bandwidth. We instantiate the cost model and predicate for MLA, but neither is MLA-specific: they apply wherever compression or sparse selection shrinks attention to small chunks (DeepSeek-V3.2, V4, and GLM-5.1 today). Extending them to a new architecture requires measuring just two coefficients: the routed payload and fetch ’s move-the-cache cost. 1. Introduction Modern LLM serving organizes KV state around prefix reuse : a tenant’s conversation builds a prefix tree, KV is cached per token, and reuse happens within one instance, one tenant, one conversation ( Kwon et al. 2023 ; Zheng et al. 2024 ) . A different reuse pattern is emerging one level up, at the provider-curated canonical content layer. A provider that pre-prefills a canonical corpus (a body of case law, public companies’ annual and quarterly reports, the trending code repositories, or a frozen documentation snapshot) into Multi-head Latent Attention (MLA) compressed-cache ( c^{\mathrm{KV}} ) form can serve any tenant whose prompt touches that content from the precomputed cache rather than re-prefilling it. The unit of reuse is no longer one conversation’s prefix but a canonical chunk shared across tenants and requests. Local prefix caching cannot capture this reuse: each request precedes the shared chunk with different content, so a contiguous prefix match breaks before reaching it, and the chunk is reusable only through cross-instance discovery by canonical id, not by extending a local cache. This regime breaks a sizing assumption. A few thousand canonical chunks fit in one accelerator’s HBM, but the hot tail grows (the actively-served slice of these corpora runs to hundreds of gigabytes of c^{\mathrm{KV}} , the full corpora orders of magnitude more) and the chunks are read by tenants spread across many serving instances. Once the canonical store exceeds a single instance’s HBM it must be partitioned across instances, and answering a request then routinely requires attending over c^{\mathrm{KV}} that lives on another instance. The same partitioning arises one scope down: a single tenant’s private corpus (an enterprise knowledge base, a large codebase, or an M&A due-diligence data room) can outgrow one instance too, reused across that tenant’s requests rather than across tenants. The agentic workload sharpens this to a single artifact: one large private document (a codebase, a contract set, a long report) pinned once as an immutable prefix and queried by many concurrent sub-agents, each forking it copy-on-write and appending only its own short suffix. The shared c^{\mathrm{KV}} then dwarfs any one agent’s appended tokens and is attended by all of them at once: the byte asymmetry above, here also a fan-in. Cross-instance attention becomes a steady-state operation, not an exception. How should it be done? The reflex (taken by essentially every cross-instance KV system to date ( Qin et al. 2025 ) ) is to move the cache : pull the remote chunk’s c^{\mathrm{KV}} over the network, splice it into the local cache, and attend locally. This reflex carries a habit over from dense attention: bring the cache to wherever the model is already running, which pays off when the moved cache is reused by a long local decode. MLA quietly changes the arithmetic for per-request cross-instance attention. Its defining feature is a narrow latent: each token’s key and value collapse into one compressed vector (the same d_{qk}{=}576 in DeepSeek-V2-Lite and the frontier V3/R1), so a single query row is only \approx 1 KB on the wire while the chunk it attends to holds thousands of equally-wide cache vectors, three orders of magnitude more data. MLA is, moreover, a frontier default rather than a niche choice: DeepSeek (V2 through the sparse-attention V3.2) ( DeepSeek-AI et al. 2024 ; DeepSeek-AI et al. 2025 ) , Kimi K2.6 ( Kimi Team et al. 2026 ; Moonshot AI 2026 ) , and the GLM family (from GLM-4.7-Flash onward, with GLM-5.1 pairing MLA with sparse attention) ( GLM-5-Team et al. 2026 ; Z.ai 2026 ) all build on MLA or a close latent-attention variant, so the byte asymmetry we exploit is broadly shared. Because our cost model depends on the model only through the wire payload q{+}p (a function of d_{qk} and d_{v} ), it instantiates to any of them from published dimensions, with DeepSeek-V2-Lite as our measured instance. When the query is the small object, the textbook move inverts: it is cheaper to route the query to the instance holding the c^{\mathrm{KV}} , compute partial attention there, and merge the small partial back. The routing primitive itself is not new; it was introduced for sharded KV and since extended to MLA and to single-job context parallelism ( Lin et al. 2024 ; Bhatia et al. 2025 ; Yang et al. 2025 ) (§ 9 ), but its economic case under MLA, across serving instances, on commodity datacenter fabric has not been established. That case is not obvious, for two reasons. First, routing a query batch is only cheap if a per-request, cross-node transfer is itself cheap, which points to device-initiated RDMA (NVSHMEM IBGDA), where the GPU issues the transfer directly with no host round trip. Yet recent transport work finds NVSHMEM IBGDA can be slower than a host-mediated path for the tiny messages of MoE all-to-all — behind a CPU proxy in NCCL GIN’s measurements, with portable engines falling back to a host proxy where the NIC lacks IBGDA ( Hamidouche et al. 2025 ; Licker et al. 2026 ) (quantified in §

Read the complete text