Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference

Vasu Shyam Anna Golubeva Quentin Anthony · 2026

Read Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference by Vasu Shyam Anna Golubeva Quentin Anthony (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26294.txt.

Description

Shyam, Golubeva, and Anthony presented tensor and sequence parallelism, a scheme that folds two splitting strategies onto one device axis. Normally weights and tokens are split along separate axes, but here each rank holds a weight shard plus a sequence shard, cutting parameter and activation memory together. Two runtime schedules handle attention and gated MLPs, trading extra communication for lower memory. Benchmarks position the method as a hardware-aware option for long-context, memory-tight training.

Catalog record

CREATORVasu Shyam Anna Golubeva Quentin Anthony
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59532
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26294

Browse the full library · About the Tower of Babel