Scaling State-Space Models on Multiple GPUs with Tensor Parallelism

Anurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi · 2026

Read Scaling State-Space Models on Multiple GPUs with Tensor Parallelism by Anurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21144.txt.

Description

State-space models handle long contexts well, but single-GPU memory, bandwidth, and latency cap their inference. Anurag Dutt, Nimit Shah, Hazem Masarani, and Anshul Gandhi scale them across GPUs in this 2026 arXiv paper via a communication-efficient tensor-parallel design for selective state-space inference. Three engineering challenges anchor the design: a state cache across prefill and decode, local recurrent updates through careful parameter partitioning, and cheaper aggregation via quantized AllReduce.

Tests on three representative models across two GPU clusters show throughput rising about one and a half to two times on two GPUs. Gains reach about two and a half to four times on four GPUs, with the largest gains at long contexts. Quantized aggregation adds a further ten to eighteen percent. Readers get a practical recipe for multi-GPU state-space serving.

Catalog record

CREATORAnurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-21144

Browse the full library · About the Tower of Babel