Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale

Jerome Ku, Eric Nguyen, David W. Romero, Garyk Brixi, Brandon Yang, Anton Vorontsov, Ali Taghibakhshi, Amy X. Lu, Dave P. Burke, Greg Brockman, Stefano Massaroli, Christopher Ré, Patrick D. Hsu, Brian L. Hie, Stefano Ermon, Michael Poli · 2025

Read Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale by Jerome Ku, Eric Nguyen, David W. Romero, Garyk Brixi, Brandon Yang, Anton Vorontsov, Ali Taghibakhshi, Amy X. Lu, Dave P. Burke, Greg Brockman, Stefano Massaroli, Christopher Ré, Patrick D. Hsu, Brian L. Hie, Stefano Ermon, Michael Poli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01868.txt.

Introduction

Jerome Ku and co-authors present new designs for large language models in this 2025 paper. Their models mix convolution steps with attention in one system. Different operators suit different jobs, such as recall of context or squeezing data down.

At the forty billion parameter scale, their models train much faster than tuned Transformers. The new StripedHyena 2 design doubles the throughput of some prior operators on H100 chips. The team co-designed the math and the hardware-aware code to get these gains. Their Evo 2 models show strength on byte-level data. The paper also covers blocked kernels and ways to split context across chips.

Catalog record

CREATORJerome Ku, Eric Nguyen, David W. Romero, Garyk Brixi, Brandon Yang, Anton Vorontsov, Ali Taghibakhshi, Amy X. Lu, Dave P. Burke, Greg Brockman, Stefano Massaroli, Christopher Ré, Patrick D. Hsu, Brian L. Hie, Stefano Ermon, Michael Poli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01868

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale Jerome Ku, Eric Nguyen, David W. Romero, Garyk Brixi, Brandon Yang, Anton Vorontsov, Ali Taghibakhshi, Amy X. Lu, Dave P. Burke, Greg Brockman, Stefano Massaroli, Christopher Ré, Patrick D. Hsu, Brian L. Hie, Stefano Ermon, Michael Poli Abstract We introduce † † footnotetext: ∗ These authors contributed equally to this work. † Current address: OpenAI. ‡ Corresponding author: [email protected]. convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression, with input-dependent convolutions and attention offering complementary performance. Second, co-designing convolution operators and hardware-aware algorithms enables efficiency gains in regimes where previous alternative architectures struggle to surpass Transformers. At the 40 billion parameter scale, we train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids. On H100 GPUs and model width 4096, individual operators in the proposed multi-hybrid StripedHyena 2 architecture achieve two-fold throughput improvement over linear attention and state-space models. Multi-hybrids excel at sequence modeling over byte-tokenized data, as demonstrated by the Evo 2 line of models. We discuss the foundations that enable these results, including architecture design, overlap-add blocked kernels for tensor cores, and dedicated all-to-all and point-to-point context parallelism strategies. \LettrineTextFont Figure 1: Scaling experiments, showing differences in perplexity and throughput of Transformers, multi-hybrids (StripedHyena 2), and other alternative operators. 1 Introduction Architecture improvements for training language models at scale can be broadly categorized into several main groups. Tweaks to the attention mechanism to reduce the size of the kv cache such as GQA, MQA, MLA, sliding window and linear attention ( Shazeer, 2019 ; Brown et al., 2020 ; Vaswani et al., 2021 ; Katharopoulos et al., 2020 ; Ainslie et al., 2023 ; Liu et al., 2024 ) . Changes to the model for numerical stability, resilience to outliers and quantization such as pre-norm, SwiGLU, QK normalization ( Zhang & Sennrich, 2019 ; Xiong et al., 2020 ; Shazeer, 2020 ) . Finally, modifications that improve model capacity or recall at longer context such as RoPE, MoE ( Shazeer et al., 2017 ; Su et al., 2024 ) . Despite the broad interest in architecture improvement, remarkably few proposals, outside of the aforementioned methods, have delivered consistent gains at scale. A different approach is to introduce new classes of input-dependent operators to the standard mix of layers (self-attention and feed-forward) layers and optimize their composition, resulting in hybrid architectures. Hybrids promise improvements on both quality and efficiency, and have been proposed in various domains and with various mixtures of operators, typically with some combination of convolution and attention ( Dai et al., 2021 ; Poli et al., 2023b ) , linear attention and attention ( Fu et al., 2022 ; Fathi et al., 2023 ; Lieber et al., 2024 ; Glorioso et al., 2024 ) , or local attention and attention ( Child et al., 2019 ; Beltagy et al., 2020 ) . In language modeling at scale, hybrids of convolutions, linear attention and attention have been validated through dedicated scaling laws ( Poli et al., 2024 ) and large-scale model releases ( Glorioso et al., 2024 ; Nguyen et al., 2024 ; Team et al., 2024 ) . Despite being a promising alternative, hybrids based on operators such as linear attention or state-space models have struggled to replace baseline Transformers as the de facto standard for language modeling due to a variety of reasons. One limitation is that these fixed-state operators realize efficiency gains only when applied to very long sequences, which is also where they drastically underperform full self-attention ( Arora et al., 2023 ; Jelassi et al., 2024 ) in quality. Compared to Transformers, these methods are generally slower in common pretraining regimes: shorter contexts with larger and wider models. Furthermore, most of these approaches have been developed with the explicit goal of matching self-attention performance on in-context recall over longer sequences, but require hybridization with self-attention to perform competitively in practice. This has introduced redundancy in architectures, as multiple operators are optimized for the same capability: in-context recall. This paper explores a fundamentally different approach. We advocate for model architecture designs that are both hybridization-aware and hardware-aware, combining different types of operators with complementary capabilities and computational costs, across a range of input and model sizes. Our approach is motivated by work on synthetics ( Akyürek et al., 2024 ) and mechanistic design ( Poli et al., 2024 ) , showing how different operators in hybrids can specialize to subtasks such as recall, compression, multi-query recall, and fuzzy recall. For example, input-dependent convolutions excel at filtering noise and performing multi-token recall, useful for modeling byte-level data, whereas attention is optimized for targeted recall of information across longer sequences. We introduce multi-hybrids , architectures that combine strengths of multiple operator types. We focus on StripedHyena 2, the first example of a convolutional multi-hybrid architecture for sequence modeling validated on a series of experiments at scale (40 billion parameters, 9 trillion tokens). StripedHyena 2 is based on three different types of input-dependent convolutional operators: (i.) short, explicitly-parametrized hyena operators that maximize hardware utilization, specializing in local multi-token recall, (ii.) medium-length, regularized hyena operators tailored to efficient modeling across hundreds of tokens, and (iii.) long, implicit hyena operators that aggregate information over the entire sequence. We describe the algorithmic foundations of convolutional multi-hybrids, focusing on architecture design, kernels, and context parallelism algorithms. As a motivating example, we will use the experiments behind the Evo 2 line of models ( Brixi et al., 2025 ) , built on top of StripedHyena 2. Evo 2 40B is a state-of-the-art foundation model for genomics, trained on byte-tokenized (nucleotide) sequences. Outline In Section 2 , we introduce the basic design ideas and describe the three primary operators behind convolutional multi-hybrids. We then discuss composition, filter grouping for improved hardware utilization, and showcase scaling at the thousand GPU and 40 billion parameter scale. In Section 3 , we focus on architecture and algorithm co-design. Using filter grouping, we adapt overlap-add algorithms ( Burrus & Parks, 1985 ) to tensor cores, introducing our implementation of a two-stage blocked kernel. We measure the performance gains at short and long context compared to efficient attention implementations (FlashAttention3 ( Shah et al., 2024 ) , SDPA) and other alternative operators such as linear attention and state-space models: Mamba2 ( Dao & Gu, 2024 ) , xLSTM ( Beck et al., 2024 ) and DeltaNet ( Yang et al., 2024 ) . In Section 4 , we develop custom context parallelism methods for the different types of convolutions in our models. We introduce both peer-to-peer and all-to-all algorithms, including new channel-pipelined variants and FFT-based methods. 2 Multi-Hybrid Model Architecture Notation Unless specified otherwise, (input) sequences are denoted with x\in\mathbb{R}^{\ell\times d} . We use subscripts to index in the time dimension, and Greek superscripts to index in the space (or width) dimension. To keep the notation compact, we also occasionally omit summation signs for repeated indices in longer tensor contractions e.g., y^{\alpha}_{t}=A^{\alpha\beta}x^{\beta}_{t} is shorthand for y^{\alpha}_{t}=\sum_{\beta}A^{\alpha\beta}x^{\beta}_{t} . 2.1 Basic Design We consider input-dependent convolutional operators

Read the complete text