Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Vasu Shyam Anna Golubeva Quentin Anthony · 2026
Read Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference by Vasu Shyam Anna Golubeva Quentin Anthony (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26294.txt.
Description
Shyam, Golubeva, and Anthony presented tensor and sequence parallelism, a scheme that folds two splitting strategies onto one device axis. Normally weights and tokens are split along separate axes, but here each rank holds a weight shard plus a sequence shard, cutting parameter and activation memory together. Two runtime schedules handle attention and gated MLPs, trading extra communication for lower memory. Benchmarks position the method as a hardware-aware option for long-context, memory-tight training.
Catalog record
| CREATOR | Vasu Shyam Anna Golubeva Quentin Anthony |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59532 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26294