Yonghao Zhuang

* · 2025

Read Yonghao Zhuang by * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18121.txt.

Introduction

Yonghao Zhuang, with eight co-authors, speeds up training of language models on very long texts by moving one heavy step to its own pool of machines. Core attention grows far faster than the rest of the model as context lengthens, which leaves some devices idle and others swamped. Their CAD design splits that step away from the main layers.

Two facts make the split work: attention holds no trainable weights, and modern kernels handle mixed batches of shards well. Token-level attention tasks are dispatched to dedicated servers and balanced like jobs. Training engineers get a cleaner way to scale long-context runs.

Catalog record

CREATOR*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Yonghao Zhuang * Abstract We present core attention disaggregation (CAD), a technique that improves long-context LLM training by disaggregating the core attention (CA) – the parameter-free softmax ⁡ ( 𝐐𝐊 ⊤ ) ​ 𝐕 \mathrm{softmax}(\mathbf{QK}^{\top})\mathbf{V} computation – and schedules it on an independent pool of resources. Existing systems co-locate core attention with other components. At long context, the quadratic growth of CA computation and near-linear growth of the rest create load imbalance – hence stragglers across data and pipeline groups. CAD is enabled by two key observations: (i) statelessness : CA has no trainable parameters and minimal transient state, so balancing reduces to scheduling compute-bound tasks; and (ii) composability : modern attention kernels sustain high utilization on fused batches of arbitrary-length token-level shards. CAD dynamically partitions the core attention computation into token-level tasks (CA-tasks), and dispatches them to a pool of devices specialized for CA computation (attention servers). It then rebatches CA-tasks to equalize CA compute across attention servers without loss of kernel efficiency. We have implemented CAD in a system called DistCA with a ping-pong scheme to completely overlap communication with compute, and in-place attention servers to improve memory utilization. On up to 512 H200 GPUs and 512K context length, DistCA improves end‑to‑end training throughput by up to 1.35 × \times , eliminates DP/PP stragglers, and maintains near‑perfect compute and memory balance. marginparsep has been altered. topmargin has been altered. marginparwidth has been altered. marginparpush has been altered. The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. Efficient Long-context Language Model Training by Core Attention Disaggregation † † footnotetext: * Equal contribution † Work done at UCSD (internship) 1 Carnegie Mellon University 2 UC SanDiego 3 MBZUAI 4 StepFun 5 UC Berkeley. Correspondence to: Eric Xing <[email protected]>, Hao Zhang <[email protected]>. 1 Introduction Figure 1: Transformer and its workload imbalance caused by core attention. Figure 2: DistCA architecture. Table 1: Compute and Memory for components in LLMs. l is the document’s number of tokens. Memory Compute CA 0 O(l^{2}) Linear O(l) O(l) MISC O(l) \approx 0 CA: core attention layer; Linear: FFN, qkvo-proj MISC: layer norm, dropout, … Recent large language model (LLM) applications show a steadily increasing demand for processing longer contexts. For instance, reasoning workloads must generate long chain-of-thoughts to yield accurate answers Guo et al. 2025 ; coding agents operate over multi-file repositories Liu et al. 2024b . To reliably support these use cases, LLMs must operate with contexts of 100K - 1M tokens at inference. Equipping an LLM with long-context capability typically requires training on datasets that include long documents in addition to ordinary (short) documents Gao et al. 2025 . A standard approach to batch documents of variable length is document packing Rae et al. 2021 ; Wang et al. 2024 : concatenate multiple documents into a fixed-size chunk and apply an attention mask to block cross-document attention. While document packing improves throughput, it leads to variance in the attention compute per chunk – therefore load imbalance – across different chunks Lin et al. 2025 . The root cause is that self-attention compute in transformers grows quadratically with sequence length, whereas the rest scales approximately linearly. Consequently, two chunks with the same total tokens can incur very different attention FLOPs depending on how documents are distributed. For example, in Figure 2 , a chunk with a single 4K-token document requires roughly 4x the attention FLOPs of a chunk packed with four 1K-token documents, even though both contain 4K tokens. This imbalance manifests as stragglers ( Figure 2 ) in large-scale distributed training in two ways ( Wang et al. 2025c ; Lin et al. 2025 ) . First, in data parallelism (DP), replicas process different chunks and synchronize at the gradient barrier; the replica with the largest attention workload stalls the others. Second, in pipeline parallelism (PP) ( Huang et al. 2019 ; Shoeybi et al. 2019 ) , pipeline stages operate on different microbatches concurrently; a microbatch with a larger attention workload makes its current stage a straggler, creating pipeline bubbles which stall the entire pipeline. In hybrid DP and PP, these effects compound, and slowdowns of 1.34-1.44x ( Wang et al. 2025c ; Lin et al. 2025 ) have been reported even under modest context lengths. One remedy is to equalize compute by assigning more tokens to devices processing shorter documents Wang et al. 2025c . This balances compute but unbalances memory, as activation footprints grow with total tokens. In the “4 \times 1K vs. 1 \times 4K” example: matching the 1 \times 4K attention FLOPs requires 12 more 1K documents on the 4 \times 1K device, raising its activation memory by 3\times . Another remedy, context parallelism (CP) ( Liu et al. 2024a ; Jacobs et al. 2023 ) , shards each document along the sequence dimension and equally distributes shards across DP replicas, which balances compute and memory, but at the cost of extra communication of KV states. Unfortunately, it cannot mitigate PP stragglers: pipeline stages hold disjoint model layers and cannot jointly compute shards of the same document; straggler microbatches still generate bubbles that stall other stages. Fundamentally, the imbalance stems from mismatched complexity between attention and the rest of the model computations, as illustrated in Table 2 . When attention and other layers are colocated and scaled together, the mismatch grows with model scale and context length, leading to severe load imbalance. This naturally leads us to disaggregate attention from the rest of the model, so the attention and non-attention components can be scaled independently to balance workload across devices. A challenge, however, is that disaggregation introduces per‑layer transfers of inputs and outputs across the attention boundary. At first glance, this communication is prohibitive and could negate its benefits. Surprisingly, we find the opposite: precisely isolating the core attention (CA)—the weightless \mathrm{softmax}(\mathbf{QK}^{\top})\mathbf{V} computation ( Figure 2 )—and applying targeted overlap and placement optimizations allows the communication to be effectively hidden in today’s long-context training workloads. Specifically, we identify two key characteristics of core attention that makes disaggregation effective. First, statelessness : CA contains no trainable parameters and stores only small per-row softmax statistics; balancing CA hence reduces to scheduling compute-bound tasks. Second, composability : CA can be partitioned at token granularity into arbitrary-length shards. Each shard, given its target tokens’ \mathbf{Q} and context tokens’ \mathbf{K} , \mathbf{V} , independently computes the output. Shards from different documents (i.e., DP replicas or pipeline stages) can be re-batched into a single high-occupancy kernel. In modern attention kernels (e.g., Flash Attention), throughput primarily depends on the aggregate tokens in the fused call rather than their document of origin. This allows partitioning and recombining shards to equalize CA compute, instead of committing to the uniform splits typically assumed by CP. These observations lead to core attention disaggregation (CAD), which disaggregates CA and schedules it independently on a pool of resources specialized for CA computation, named attention servers . Attention servers accept core attention tasks—the CA computation of arbitrarily partitioned document shards—as compute requests. The

Read the complete text