Caracal: Causal Architecture via Spectral Mixing
Bingzheng Gan, Tianyi Zhang, Yusu Li, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu · 2026
Read Caracal: Causal Architecture via Spectral Mixing by Bingzheng Gan, Tianyi Zhang, Yusu Li, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00292.txt.
Introduction
Bingzheng Gan and 6 co-authors introduce Caracal, a language model architecture that replaces attention with Fourier mixing. Attention grows costly very fast as sequences lengthen, and position codes add their own limits. Caracal instead mixes information across a sequence with the Fast Fourier Transform, a classic signal processing tool, at a much gentler cost growth.
A frequency-domain masking trick, built from uneven padding and trimming, keeps the model causal so it can generate text step by step. Unlike some rival efficient designs, Caracal uses only standard library operations, which makes it portable across hardware. In evaluations it performed competitively against Transformer and state-space baselines. The authors present it as a simple, scalable path for modeling long sequences.
Catalog record
| CREATOR | Bingzheng Gan, Tianyi Zhang, Yusu Li, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00292
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Caracal: Causal Architecture via Spectral Mixing Bingzheng Gan, Tianyi Zhang, Yusu Li, Jing Huang, Wei Shi, Yangkai Ding, Tao Yu Abstract The scalability of Large Language Models to long sequences is hindered by the quadratic cost of attention and the limitations of positional encodings. To address these, we introduce Caracal , a novel architecture that replaces attention with a parameter-efficient, 𝒪 ( L log L ) \mathcal{O}(L\log L) Multi-Head Fourier (MHF) module. Our contributions are threefold: (1) We leverage the Fast Fourier Transform (FFT) for sequence mixing, inherently addressing both bottlenecks mentioned above. (2) We apply a frequency-domain causal masking technique that enforces autoregressive capabilities via asymmetric padding and truncation, overcoming a critical barrier for Fourier-based generative models. (3) Unlike efficient models relying on hardware-specific implementations (e.g., Mamba), we uses standard library operators. This ensures robust portability, eliminating common deployment barriers. Evaluations demonstrate that Caracal performs competitively with Transformer and SSM baselines, offering a scalable and simple pathway for efficient long-sequence modeling. Code is available in Appendix E . 1 Introduction While the Transformer ( Vaswani et al. 2017 ) remains the standard for sequence modeling, its reliance on attention introduces two structural bottlenecks. The first is computational complexity: operations scale quadratically with input sequence length L , i.e., \mathcal{O}(L^{2}) , making long sequences prohibitively expensive. The second is permutation equivariance; attention is insensitive to token order, necessitating external positional encodings. These encodings, whether absolute or relative, often tether performance to training context lengths, causing instability during extrapolation and imposing a context window limit. One major line of research, the Enhancement Paradigm , aims to mitigate these issues by modifying the attention framework. To address the computational cost, sparse attention methods reduce complexity by introducing fixed connectivity patterns, as seen in Longformer ( Beltagy et al. 2020 ) and BigBird ( Zaheer et al. 2020 ) , or by employing adaptive, content-based mechanisms like those in Reformer ( Kitaev et al. 2020 ) and Routing Transformer ( Roy et al. 2021 ) . While effective, these methods risk information loss. Concurrently, sophisticated relative positional encodings like RoPE ( Su et al. 2024 ) , YaRN ( Peng et al. 2024 ) , and ALiBi ( Press et al. 2022 ) , which encode relative distances or apply distance-aware biases, have dramatically improved length extrapolation. However, the growing complexity of these techniques shows they are sophisticated compensations for a mechanism lacking a built-in sequence concept, motivating the search for a more fundamental alternative. A more radical approach, the Replacement Paradigm , replaces the attention mechanism with more efficient alternatives. State Space Models (SSMs), particularly the recent Mamba architecture ( Gu & Dao 2023 ; Dao & Gu 2024 ) , have emerged as a powerful contender. Mamba achieves linear-time complexity and state-of-the-art performance, but its efficiency heavily relies on hardware-specific custom CUDA kernels, hindering portability, modification, and broad adoption. Another prominent direction within this paradigm leverages the Fourier Transform, offering an appealing \mathcal{O}(L\log L) complexity for global token mixing ( Lee-Thorp et al. 2022 ) . However, these Fourier-based models have historically struggled with generative tasks. Their primary flaw lies in the difficulty of enforcing causality —a strict requirement for autoregressive decoding—within the frequency domain, a challenge that has relegated them to encoder-only or non-autoregressive applications. In this work, we argue that a solution can be found that is both algorithmically elegant and hardware-agnostic. We introduce Caracal , a novel architecture that directly confronts the foundational issues of attention. By replacing the global attention mechanism with our Multi-Head Fourier (MHF) module, Caracal inherently resolves the quadratic complexity and the need for positional encodings. Furthermore, by applying a novel causal masking technique that operates purely in the frequency domain, we overcome the critical causality barrier that has hindered previous Fourier-based models. In practice, to maximize local feature extraction precision, we strategically retain a small proportion of attention layers, constraining them to a sliding window mechanism. Crucially, this design choice preserves our architecture’s fundamental advantages: the MHF layers inherently provide global positional information, eliminating the need for explicit positional encodings even in the attention layers, while the fixed-size sliding window of attention ensures the overall computational complexity remains efficient and scalable. Our contributions are as follows: • A Novel Autoregressive Fourier Module: We propose the Multi-Head Fourier (MHF) module, which mixes token information via a gated element-wise product in the frequency domain, directly replacing the attention layer in a Transformer. • Frequency-Domain Causal Masking: We apply a technique that enforces causality entirely in the frequency domain using asymmetric padding and truncation. This addresses a key obstacle that has hindered the application of Fourier models to generative tasks. • A Position-Encoding-Free Architecture: Our model obviates the need for explicit positional encodings because the Fourier Transform inherently captures position through its sinusoidal basis functions. This design is theoretically advantageous for length extrapolation. • Solid Validation: We show that Caracal achieves performance competitive with state-of-the-art Transformer and SSM baselines on diverse benchmarks. Crucially, it strikes a unique balance: while its \mathcal{O}(L\log L) complexity is slightly above the \mathcal{O}(L) scaling of SSMs, it remains significantly more efficient than the \mathcal{O}(L^{2}) of Transformers, all while bypassing the hardware-dependent optimization hurdles and implementational complexities typically inherent to SSMs. 2 Related Work To address the limitations of Transformers, researchers have sought efficient alternatives, primarily categorized into State Space Models (SSMs) rooted in continuous control theory, Linear Recurrent Models that optimize hidden state update rules for associative memory, and Spectral Methods which leverage the efficiency of the Fast Fourier Transform (FFT). 2.1 State Space Models (SSMs) SSMs have recently emerged as a powerful efficient replacement for attention. Foundational work like the S4 ( Gu et al. 2022b ) and S4D ( Gu et al. 2022a ) demonstrates the potential for modeling long-range dependencies by leveraging a formulation computable as a global convolution. However, the time-invariant nature of S4’s state matrices limits its ability to perform content-aware reasoning. To address this, subsequent architectures introduce gating mechanisms and data-dependent primitives. H3 ( Fu et al. 2023b ) introduces multiplicative gating to mimic linear attention. Hyena ( Poli et al. 2023 ) employs implicit long convolutions by generating filters via a small MLP acting on positional encodings, leveraging FFTs for sub-quadratic processing. While effective, Hyena’s filters are strictly position-based (dependent on t ), whereas Caracal ’s filters are content-based (generated dynamically from input tokens x ), enabling direct semantic interactions akin to attention. Most recently, Mamba ( Gu & Dao 2023 ) and Mamba-2 ( Dao & Gu 2024 ) achieve state-of-the-art performance via selective, input-dependent state updates. While improving expressivity, this selectivity sacrifices the parallel convolutional form inherent in earlier SSMs, necessitating complex hardware-aware scans or specialized block-partitioning to maintain efficiency. Such SSD-style models often involve a high conceptual barrier, requiring deep expertise in state-space duality and structured matrix decomposition. This mathematical complexity can lead to a ”black-box” nature that makes it challenging to modify internal logic without risking architectural breakdowns. Furthermore, achieving peak performance in modern SSMs often requires manual tuning of block sizes or reliance on custom CUDA kernels, hindering portability. In contrast, Caracal provides superior structural clarity and flexibility by leveraging standard FFT operators, ensuring universal compatibility