Model Parallelism With Subnetwork Data Parallelism

Vaibhav Singh, Zafir Khalid, Edouard Oyallon, Eugene Belilovsky · 2025

Read Model Parallelism With Subnetwork Data Parallelism by Vaibhav Singh, Zafir Khalid, Edouard Oyallon, Eugene Belilovsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09029.txt.

Introduction

Vaibhav Singh and co-authors introduce a new way to train large networks in a 2025 arXiv paper. Pre-training at scale strains memory and needs costly communication between machines. Their framework, Subnetwork Data Parallelism, splits a model into subnetworks trained on separate workers.

Workers never exchange activations, which cuts communication sharply. Two masking regimes are studied: one sparsifies only the backward pass, the other also thins the forward pass. Subnetworks can be built at neuron level or block level. Tests ranged from a billion-parameter language model to an image network. Memory per device fell by up to sixty percent while performance held or improved.

Catalog record

CREATORVaibhav Singh, Zafir Khalid, Edouard Oyallon, Eugene Belilovsky
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09029

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Model Parallelism With Subnetwork Data Parallelism Vaibhav Singh, Zafir Khalid, Edouard Oyallon, Eugene Belilovsky Abstract Distributed pre-training of large models at scale often imposes heavy memory demands on individual nodes and incurs significant intra-node communication costs. We propose a novel alternative approach that reduces the memory requirements by training small, structured subnetworks of the model on separate workers. Unlike pipelining, our method avoids inter-node activation communication and maintains bandwidth requirements that are comparable to or lower than standard data parallel communication schemes based on all-reduce. We evaluate two subnetwork construction strategies guided by the principle of ensuring uniform representation of each parameter across the distributed training setup. Our results show that the stochastic block dropping technique consistently outperforms the width-wise subnetwork construction previously explored in federated learning. We empirically attribute this superior performance to stronger gradient alignment in subnetworks that retain blocks having skip connections. Preliminary experiments highlight the promise of our approach, achieving a 𝟐𝟎 ​ – ​ 𝟒𝟎 % \boldsymbol{20–40}\% reduction in memory usage without any loss in performance. † † affiliationnotice: Equal contribution 1 Introduction The rapid scaling of deep neural networks has led to unprecedented progress across a wide range of domains, from computer vision ( He et al. 2016a ; Radford et al. 2021 ; Oquab et al. 2023 ; Kirillov et al. 2023 ; Shang et al. 2024 ) to natural language processing ( Bommasani et al. 2021 ; Achiam et al. 2023 ; Touvron et al. 2023 ; Zhao et al. 2023a ) . Training such large models has necessitated distributed strategies like data parallelism ( Li et al. 2020 ) and model parallelism ( Shazeer et al. 2018 ; Shoeybi et al. 2019 ; Huang et al. 2019 ) , each with trade-offs. Data parallelism, especially in the form of Distributed Data Parallel (DDP) ( Li et al. 2020 ) , replicates the model on each GPU and synchronizes gradients after each backward pass. While simple and widely adopted, it incurs high memory overhead due to full model replication and high communication cost during synchronization. Model parallelism (e.g., GPipe ( Huang et al. 2019 ) ) mitigates memory issues by splitting layers across devices but requires expensive high-bandwidth interconnects to communicate activations. Unlike data parallelism, where several techniques effectively reduce communication costs ( Douillard et al. 2023a ; Wang et al. 2023 ) , reducing the bandwidth constraints when communicating activations is an open problem. Additionally, classic pipeline-based approaches often suffer from inefficiencies due to idle waiting (pipeline bubbles). In this work, we explore Subnetwork Data Parallelism as a strategy to reduce per-node memory usage by fully distributing the training of model sub-components across devices. Unlike pipelining, which divides computation across sequential stages, our approach assigns each worker a subnetwork , defined as a structurally complete portion of the model with a full path from input to loss, enabling independent gradient computation without exchanging activations. Each worker optimizes only its subset of parameters and synchronizes overlapping parameters through averaging at each step. Instead of replicating or fully sharding the model, we distribute these subnetworks across nodes, so that each device hosts only a fraction of the full model. Each subnetwork is trained independently, with synchronization performed via parameter averaging to form a unified model. Since each node hosts only a fraction of the full model, overall memory usage is significantly reduced. Intra-node parallelism follows standard data parallelism with parameter-level communication, and the approach is compatible with existing systems-level model parallelism techniques. Subnetwork training draws on two key insights: (i) large models often contain significant parameter redundancy, making it possible to estimate the loss with subnetworks; and (ii) partial training acts as a form of regularization, improving generalization. By training overlapping submodels and synchronizing only shared parameters, our framework balances scalability and efficiency. This yields a substantial reduction in memory footprint, as each GPU hosts only a subnetwork rather than the full model. By relaxing the need for full replication and reducing synchronization overhead, our approach improves both memory efficiency. We formalize this approach as a distributed training framework in which each worker optimizes a fixed subset of model parameters, referred to as its subnetwork . Each parameter is assigned to a fixed set of P\leq N workers, ensuring that all parameters are updated by at least one worker. Following each local update, parameter replicas are averaged and broadcasted to maintain coherence across the global model. This strategy, which we term Subnetwork Data Parallelism , avoids full model replication and pipeline dependencies, enabling efficient training on memory-constrained accelerators. • We propose a distributed training paradigm— Subnetwork Data Parallelism , based on overlapping parameter assignment and averaging, enabling memory efficient training. • We explore two subnetwork construction strategies: (i) selecting subsets of neurons or channels, and (ii) removing entire layers (blocks) from the network. • We demonstrate that our approach achieves competitive performance on image classification tasks, while substantially reducing per-device memory usage and synchronization overhead. 2 Related Work We now review the related work in model parallelism and in federated learning. Pipeline parallelism Pipeline parallelism addresses memory bottlenecks by dividing the model itself across devices. In ( Huang et al. 2019 ; Rivaud et al. 2024 ) , the model is partitioned layer-wise across multiple devices, and micro-batches are pipelined to maximize hardware utilization. Mesh-TensorFlow ( Shazeer et al. 2018 ) and Megatron-LM ( Shoeybi et al. 2019 ) explore intra-layer tensor-splitting, where weights and activations are sharded across devices. While these methods allow overcoming memory constraints, they require multiple well-connected accelerators and also experience pipeline bubbles and load imbalance. Another model parallelism direction has considered parallel learning of layers through auxiliary local losses ( Belilovsky et al. 2020 ) . Fully Sharded and Zero Redundancy Approaches. To address memory inefficiencies in data parallelism, recent advances such as Fully Sharded Data Parallel (FSDP) ( Zhao et al. 2023b ) , and ZeRO ( Rajbhandari et al. 2020 ) decompose model parameters, gradients, and optimizer states across devices. These methods significantly reduce the per-device memory footprint and are well-supported by frameworks like DeepSpeed ( dee 2020 ) . While these techniques significantly reduce per-device memory usage, they still involve substantial inter-device communication, particularly during gradient synchronization, which can lead to increased communication overhead and latency. SWARM Learning SWARM ( Ryabinin et al. 2023 ) learning attempts to address model parallelism and the constraints of individual GPUs by having a number of devices available at each stage of pipeline parallelism while efficiently routing samples through them. This is fundamentally different than the approach described here as it still requires communicating activations across potentially low-bandwidth links, while subnetwork training focuses on reducing all communication to parameters or gradients and maintaining only data parallelism across nodes. Federated Learning and Dropout-Based Subnetwork Training. Federated learning frameworks ( Konečný et al. 2016 ) train models across decentralized data sources, often focusing on tackling the associated non-iid problem. Several works have considered training subnetworks per device in a federated setting ( Caldas et al. 2018 ; Horvath et al. 2021 ; Guliani et al. 2022 ; Wen et al. 2022 ; Alam et al. 2022 ) . However, with significantly different goals, context, and methodology. Specifically, these works focus on

Read the complete text