Scaling State-Space Models on Multiple GPUs with Tensor Parallelism
Anurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi · 2026
Read Scaling State-Space Models on Multiple GPUs with Tensor Parallelism by Anurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21144.txt.
Description
State-space models handle long contexts well, but single-GPU memory, bandwidth, and latency cap their inference. Anurag Dutt, Nimit Shah, Hazem Masarani, and Anshul Gandhi scale them across GPUs in this 2026 arXiv paper via a communication-efficient tensor-parallel design for selective state-space inference. Three engineering challenges anchor the design: a state cache across prefill and decode, local recurrent updates through careful parameter partitioning, and cheaper aggregation via quantized AllReduce.
Tests on three representative models across two GPU clusters show throughput rising about one and a half to two times on two GPUs. Gains reach about two and a half to four times on four GPUs, with the largest gains at long contexts. Quantized aggregation adds a further ten to eighteen percent. Readers get a practical recipe for multi-GPU state-space serving.
Catalog record
| CREATOR | Anurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-21144