PipeWeaver: Addressing Data Dynamicity in Large Multimodal Model Training with Dynamic Interleaved Pipeline
Zhenliang Xue 1 Hanpeng Hu 2 Xing Chen 2 Yimin Jiang ∗ Yixin Song 1,3 Zeyu Mi 1,3 Yibo Zhu 2 Daxin Jiang 2 Yubin Xia 1 Haibo Chen 1 · 2025
Read PipeWeaver: Addressing Data Dynamicity in Large Multimodal Model Training with Dynamic Interleaved Pipeline by Zhenliang Xue 1 Hanpeng Hu 2 Xing Chen 2 Yimin Jiang ∗ Yixin Song 1,3 Zeyu Mi 1,3 Yibo Zhu 2 Daxin Jiang 2 Yubin Xia 1 Haibo Chen 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14145.txt.
Introduction
Zhenliang Xue and co-authors present DIP in 2025, a scheduling framework for training large multimodal models. Note that the catalog title differs from the text's own title, and the text governs here. Such models take flexible mixes of image, text, and other inputs, but training efficiency suffers from imbalance.
DIP splits computation by modality into dedicated pipeline segments to balance workloads. Input data are also split into finer, modality-specific pieces. Schedules are generated on idle processors during training, so no stall occurs. Tests on five models, up to ninety-four billion parameters, showed up to ninety-seven percent higher throughput.
Catalog record
| CREATOR | Zhenliang Xue 1 Hanpeng Hu 2 Xing Chen 2 Yimin Jiang ∗ Yixin Song 1,3 Zeyu Mi 1,3 Yibo Zhu 2 Daxin Jiang 2 Yubin Xia 1 Haibo Chen 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14145
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PipeWeaver: Addressing Data Dynamicity in Large Multimodal Model Training with Dynamic Interleaved Pipeline Zhenliang Xue 1 Hanpeng Hu 2 Xing Chen 2 Yimin Jiang ∗ Yixin Song 1,3 Zeyu Mi 1,3 Yibo Zhu 2 Daxin Jiang 2 Yubin Xia 1 Haibo Chen 1 Abstract . Large multimodal models (LMMs) have demonstrated excellent capabilities in both understanding and generation tasks with various modalities. While these models can accept flexible combinations of input data, their training efficiency suffers from two major issues: pipeline stage imbalance caused by heterogeneous model architectures, and training data dynamicity stemming from the diversity of multimodal data. In this paper, we present PipeWeaver, a dynamic pipeline scheduling framework designed for LMM training. The core of PipeWeaver is dynamic interleaved pipeline , which searches for pipeline schedules dynamically tailored to current training batches. PipeWeaver addresses issues of LMM training with two techniques: adaptive modality-aware partitioning and efficient pipeline schedule search within a hierarchical schedule space. Meanwhile, PipeWeaver utilizes SEMU (Step Emulator), a training simulator for multimodal models, for accurate performance estimations, accelerated by spatial-temporal subgraph reuse to improve search efficiency. Experiments show that PipeWeaver can enhance LMM training efficiency by up to 97.3% compared to state-of-the-art systems, and demonstrate excellent adaptivity to LMM training’s data dynamicity. 1. Introduction Transformer-based models ( transformer-nips17 ; qwen2-5-arxiv25 ; deepseek-v3-arxiv25 ; llama3-arxiv24 ) have demonstrated remarkable capabilities in multimodal understanding, reasoning, and generation, establishing themselves as foundational architectures for next-generation large multimodal models (LMMs) ( qwen2.5-vl-arxiv25 ; gpt4o-arxiv24 ; claude-3-2024 ; llama3-arxiv24 ; step-audio-arxiv25 ; step-t2v-arxiv25 ; gemini-3-2025 ; dsv-arxiv25 ) . Modern LMMs integrate multiple modality modules , including encoders, decoders, and the backbone model connected via modality adapters. This architecture enables flexible and seamless processing of interleaved modality data (e.g., text, images, video), thereby supporting diverse task paradigms, including multimodal document understanding ( mlongdoc-arxiv24 ) and multi-turn dialogs ( gpt4o-image-generation ) . However, this flexibility makes training LMMs a challenging task due to the irregular and fluctuating execution latencies across different modality modules and data batches, introducing the unique challenge of dynamic imbalance . This problem arises from two interrelated factors: • Pipeline Stage Imbalance: LMMs exhibit architectural heterogeneity due to diverse modality modules with distinct parameter shapes and operator types (e.g., attention, GEMM, convolution), which creates skewed computational workloads and varying memory access patterns, complicating the partitioning of model execution into balanced pipeline stages. As a result, even with exhaustive enumeration of all possible splits of model layers, the “optimal” partitioning still incurs a 22.8% pipeline bubble overhead for a 37B-parameter LMM, mainly due to the significant discrepancy between heterogeneous layers that hinders perfectly balanced pipeline stage partitioning. • Training Data Dynamicity: This inherent heterogeneity is further compounded by the diversity of multimodal training data, which consists of large-scale, diverse datasets with highly variable modality distributions across data batches. This data dynamicity makes one-fits-all pipeline schedules infeasible for LMM training. Despite data packing techniques ( wlb-llm-arxiv25 ; dynapipe-eurosys24 ) designed to produce more balanced datasets, computational imbalance persists, with the largest data sample incurring 4.15\times greater computational load than the smallest in our experiment. Such disparities induce substantial workload fluctuations between data batches, necessitating adaptive pipeline scheduling. Current distributed training systems fail to address the dynamic imbalance problem, as they either lack awareness of pipeline stage imbalance during partitioning or employ static pipeline schedules that ignore training data dynamicity. Megatron-LM ( megatron-lm-arxiv20 ) , the state-of-the-art framework for training unimodal large language models, focuses solely on model architectures comprised of repeated homogeneous transformer layers, making its design inadequate for heterogeneous modality modules. Recent training systems for multimodal models like Spindle ( spindle-asplos25 ) and Optimus ( optimus-arxiv24 ) , rely on static pipeline schedules specialized for fixed sets of training tasks, overlooking the dynamic nature of multimodal data and resulting in suboptimal pipeline performance. A promising solution to this issue is dynamically searching for pipeline schedules tailored to the content of subsequent data batches. However, dynamic pipeline scheduling faces a critical trade-off between search efficiency and estimation accuracy. First, the search process must complete within a tight time budget matching the latency of one training iteration (e.g., 10 seconds). This demands a carefully designed search space for rapid pipeline schedule exploration. Second, the search process requires frequent performance estimations for candidate pipeline schedules. Existing training schedulers employ oversimplified coarse-grained cost models to reduce estimation overhead. For example, Alpa’s cost model assumes uniform microbatch latencies ( alpa-osdi22 ) , ignoring dynamic imbalance and yielding inaccurate predictions for LMM workloads. While white-box modeling simulators ( echo-arxiv24 ; proteus-tpds24 ; simai-nsdi25 ) can achieve accurate performance estimation through detailed operator-level simulation, they necessitate full-model re-simulation whenever workloads are updated. This process incurs prohibitive overhead (e.g., minutes per simulation), significantly slowing down the search process. In this paper, we propose the dynamic interleaved pipeline and a dynamic multimodal pipeline scheduling framework named PipeWeaver. Our key observation is that modality-specific pipeline stage partitioning and microbatch splitting dissect mixed-modal training data into isolated single-modal components, which enables independent tuning to achieve globally balanced execution latencies across both modality modules and data microbatches. Based on this observation, we introduce modality-aware partitioning , which individually splits pipeline executions and data microbatches of each modality module into balanced pipeline segments (i.e., sequences of pipeline stages, defined in § 5 ) to tackle the issue of pipeline stage imbalance. To address training data dynamicity, PipeWeaver searches for efficient pipeline schedules that adaptively interleave pipeline segments from different modality modules according to the content of current training batch. We design a hierarchical schedule space with efficient exploration of pipeline schedules via tailored search algorithms. Leveraging the inherent hierarchy of modality modules, pipeline stages, and model layers, we develop specialized search strategies that align with the scheduling properties of each level. Concurrently, we implement a multimodal training simulator named SEMU (Step Emulator) that accelerates performance estimation during pipeline schedule exploration through spatial-temporal subgraph reuse . This mechanism identifies recurrent spatial computation patterns by clustering adjacent operators (e.g., nodes within a pipeline stage) into reusable subgraphs, which can be reused across different simulator invocations, effectively enhancing search efficiency while preserving estimation accuracy. We implement PipeWeaver on top of Megatron-LM, the state-of-the-art distributed training framework for transformer-based large language models. In addition to the scheduling algorithms, we augment Megatron-LM with a reconfigurable pipeline mechanism to support the deployment of dynamic interleaved pipeline. Experiments demonstrate PipeWeaver’s superior performance to baseline training systems with up to 97.3% performance improvement. Furthermore, PipeWeaver shows excellent adaptivity to dynamic workloads of LMM training, and manages to maintain maximal utilization of hardware resources throughout the training process. Contributions. We make the following contributions in designing PipeWeaver and SEMU : • We propose dynamic interleaved pipeline to address the issue of dynamic imbalance, and present PipeWeaver, a dynamic pipeline scheduling framework for LMM training. • We design a hierarchical schedule space for rapid pipeline schedule search, adaptively optimizing the training pipeline under dynamic workloads. • We implement SEMU for fast and accurate performance estimations of multimodal training workloads. • We evaluate PipeWeaver against three state-of-the-art training systems on five LMM models, demonstrating improvement up to 97.3% in training efficiency. 2. Dynamic Imbalance Characterization Figure 1. Comparison between CLIP-based models and LMMs.