BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
Ting Sun † , Junjie Zhang † , Xiao Yan ‡ , Songxin Zhang † , Zhuoyang Song † Jingyi Xi † , Zunyao Mao † , Bingyi Jing § , Jiaxing Zhang † ∗ , Zejian Xie † ∗ · 2026
Read BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training by Ting Sun † , Junjie Zhang † , Xiao Yan ‡ , Songxin Zhang † , Zhuoyang Song † Jingyi Xi † , Zunyao Mao † , Bingyi Jing § , Jiaxing Zhang † ∗ , Zejian Xie † ∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09994.txt.
Introduction
Sun and nine colleagues built BatchWeave, an object-store-native data plane for distributed large model training. Modern training turned data loading from a fixed layer into a live part of training. Colocated dataloaders offer no failure cover, while message-queue dataloaders use a record-offset model that cannot express batch semantics.
BatchWeave runs batch publication, recovery, and lifecycle with versioned manifests and conditional writes. Its Transactional Global Batch adds training consistency to ACID storage: all-rank visibility, ordered steps, and exactly-once recovery. Recovery and retention live in storage itself. A Decentralized Adaptive Commit algorithm keeps intake steady as the list grows, with no talk between producers. On 64 GPUs, BatchWeave beat colocated loader speed with full failure cover, beat Kafka on intake, and cut reader lag below Kafka's.
Catalog record
| CREATOR | Ting Sun † , Junjie Zhang † , Xiao Yan ‡ , Songxin Zhang † , Zhuoyang Song † Jingyi Xi † , Zunyao Mao † , Bingyi Jing § , Jiaxing Zhang † ∗ , Zejian Xie † ∗ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09994
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training Ting Sun † , Junjie Zhang † , Xiao Yan ‡ , Songxin Zhang † , Zhuoyang Song † Jingyi Xi † , Zunyao Mao † , Bingyi Jing § , Jiaxing Zhang † ∗ , Zejian Xie † ∗ Abstract . Modern Large Foundation Model (LFM) training has transformed the data pipeline from a static ingestion layer into a dynamic component that must co-evolve with the training process. Existing systems are ill-equipped: colocated dataloaders offer no failure isolation, while message queue-based disaggregated dataloaders operate on a record/offset abstraction that cannot express the batch-level semantics required by distributed training. We present BatchWeave, an object-store-native training data plane for distributed LFM training. BatchWeave uses versioned manifests and conditional object writes to coordinate batch publication, recovery, and lifecycle management. First, it introduces the Transactional Global Batch (TGB), which builds on versioned-manifest ACID storage semantics and extends them with training-specific consistency, including atomic all-rank batch visibility, a globally ordered step sequence, checkpoint-aligned lifecycle management, and end-to-end exactly-once recovery. Second, it realizes recovery and retention directly in the storage layer, by durably persisting producer state through the commit protocol and tying reclamation to distributed checkpoint state. Third, its Decentralized Adaptive Commit (DAC) algorithm sustains stable ingestion throughput as the manifest grows, without any inter-producer communication. Evaluations on large-scale multimodal pre-training and SFT workloads using 64 GPUs show that BatchWeave outperforms colocated dataloader throughput while providing full failure isolation, outperforms Apache Kafka in ingestion throughput, and achieves lower consumer read latency than Kafka. 1. Introduction Data loading has long been a well-studied component of model training. In conventional deep learning workloads, the data pipeline is straightforward: a static, pre-processed dataset is shuffled offline, partitioned into fixed-size batches, and streamed to the training process at a predictable rate. Systems such as tf.data ( 26 ) and MosaicML Streaming ( 25 ) are designed around this model, optimizing for throughput and shuffle quality over a fixed corpus. To prevent preprocessing from bottlenecking GPU utilization, disaggregated variants offload preprocessing to independent worker pools: tf.data service ( 5 ) and Cachew ( 12 ) run preprocessing on separate CPU nodes and deliver batches to training ranks via RPC, while CoorDL ( 23 ) coordinates cache sharing across colocated training jobs. These systems improve resource efficiency, but all share a common assumption: the dataset is static, batch membership is determined before training begins, and batches are ephemeral delivery artifacts that are discarded once consumed. Large Foundation Model (LFM) training breaks these assumptions. The data pipeline is no longer a passive feeder of a fixed corpus but a dynamic, runtime-coupled component that must co-evolve with the training process. It is dynamic in three aspects that existing systems are not designed to handle. • Data volume. In multimodal pre-training and supervised fine-tuning with video or interleaved image-text data, raw inputs undergo heavy runtime preprocessing before reaching the optimizer. Video decoding, frame extraction, resolution rescaling, and token packing all introduce data expansion whose volume depends on the input content and current model configuration. Intermediate data can exceed the raw dataset size by one to three orders of magnitude. This unpredictable expansion makes static resource allocation infeasible and demands that preprocessing be decoupled from training, with elastic storage to absorb runtime-materialized data. • Batch membership. In workloads that use online token packing or model-dependent sample selection (such as long-context continued pre-training or reinforcement learning from human feedback), the preprocessing pipeline itself runs at training time, and the exact composition of each batch is determined by its output. Batch boundaries are known only after preprocessing completes; no static partitioning scheme can capture them in advance. The data plane must therefore treat batch membership as a runtime artifact and provide a mechanism to expose complete batches atomically to all training ranks. • Checkpoint alignment. Modern LFM pipelines interleave pre-training, fine-tuning, and reinforcement learning stages, each of which may require rolling back to an earlier checkpoint or replaying a specific batch sequence. Rather than a form of dynamism per se, this reflects a requirement for persistence and versioning: the data plane must maintain a durable, checkpoint-aligned batch history so that any live checkpoint can deterministically recover the exact data it consumed. Ephemeral delivery systems cannot provide this guarantee. These requirements call for a training-aware data plane that understands the semantic structure of training steps, not one that merely transports individual records. Existing solutions do not meet this bar. Colocated dataloaders provide natural semantic alignment but no failure isolation: a preprocessing crash stalls the entire job. They are also subject to structural resource contention, where preprocessing threads share CPU cycles and memory bandwidth with the training process on the same node, imposing a throughput ceiling that in-rank engineering cannot overcome under heavy preprocessing workloads. Disaggregated dataloaders with centralized services (tf.data service, Cachew) decouple preprocessing from training and support elasticity, but deliver batches ephemerally via RPC with no persistent history and no batch-level atomicity guarantees. Neither category can express the training-aware semantics that LFM workloads require: a batch spanning multiple dynamically produced objects must be either fully visible or fully invisible to all data-parallel ranks at once, and its membership must be recoverable across failures and rollbacks. Object storage is a natural substrate for a disaggregated training data plane. Its append-only write model, elastic scalability, and decentralized access require no dedicated ingestion-service provisioning. Inspired by versioned table architectures ( 3 ; 35 ; 19 ) , we propose to govern data visibility through versioned metadata manifests rather than record offsets, separating the act of writing data from the act of exposing it to consumers. A new training job requires only a fresh namespace prefix, with no partition assignment and no cold-start overhead. Object storage alone, however, provides only single-object atomicity: it offers no batch-level transactional semantics and no training-progress-aware lifecycle management. More fundamentally, a central coordinator is structurally misaligned with LFM training: preprocessing pools are elastic, training jobs require flexible scaling of producers and consumers independently, and any server-side component introduces a new failure domain and provisioning overhead orthogonal to the job. BatchWeave’s design principle is therefore that object storage and versioned manifests should be the shared substrate for publication, recovery, and lifecycle management. Doing so requires solving problems at three levels. • Concurrent commit throughput. Decentralized writes rely on Optimistic Concurrency Control (OCC) over a versioned manifest. Under high producer concurrency and a monotonically growing manifest, the system must balance commit conflicts, commit overhead, and ingestion throughput with no inter-producer communication. • Batch-level semantics. The system must guarantee atomic visibility of each training batch across all data-parallel ranks, a globally consistent batch ordering, and exactly-once semantics under producer failures and consumer restarts. • Checkpoint-coupled lifecycle. Checkpoint-based rollbacks and cross-run reuse require a versioned batch history, while training-progress-aligned reclamation is needed to bound storage costs without a persistent coordination service. We present BatchWeave , an object-store-native training data plane. Its core abstraction is the Transactional Global Batch (TGB) , which treats each training batch as a first-class persistent entity with atomicity, durability, and globally consistent ordering guarantees. BatchWeave materializes TGBs as versioned, manifest-referenced structures on object storage, decoupling data production from training consumption while preserving the semantic boundaries