Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention
Andrew Kiruluta, Priscilla Burity, Samantha Williams · 2025
Read Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention by Andrew Kiruluta, Priscilla Burity, Samantha Williams (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08801.txt.
Introduction
Andrew Kiruluta, Priscilla Burity, and Samantha Williams propose replacing attention in transformer models in this two thousand twenty-five paper. Self-attention handles long-range dependencies well but its cost grows quadratically with sequence length, which strains long inputs and small machines. Their Learnable Multi-Scale Wavelet Transformer swaps dot-product attention for a learnable multi-scale Haar wavelet module. Wavelets naturally capture both fine local detail and broad global context at multiple resolutions.
Unlike fixed wavelet transforms, the module's scale-specific coefficients are learned end-to-end, letting the model adapt its decomposition to the task. The authors give the full mathematical formulation and an architectural diagram. On a standard machine translation benchmark they compare against a self-attention baseline using translation quality and accuracy metrics. They also highlight the design's linear scaling and the interpretability gained from visualizing the learned wavelet coefficients.
Catalog record
| CREATOR | Andrew Kiruluta, Priscilla Burity, Samantha Williams |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention Andrew Kiruluta, Priscilla Burity, Samantha Williams Abstract Transformer architectures, underpinned by the self-attention mechanism, have achieved state-of-the-art results across numerous natural language processing (NLP) tasks by effectively modeling long-range dependencies. However, the computational complexity of self-attention, scaling quadratically with input sequence length, presents significant challenges for processing very long sequences or operating under resource constraints. This paper introduces the Learnable Multi-Scale Wavelet Transformer (LMWT), a novel architecture that replaces the standard dot-product self-attention with a learnable multi-scale Haar wavelet transform module. Leveraging the intrinsic multi-resolution properties of wavelets, the LMWT efficiently captures both local details and global context. Crucially, the parameters of the wavelet transform, including scale-specific coefficients, are learned end-to-end during training, allowing the model to adapt its decomposition strategy to the data and task. We present the detailed mathematical formulation of the learnable Haar wavelet module and its integration into the transformer framework, supplemented by an architectural diagram. We conduct a comprehensive experimental evaluation on a standard machine translation benchmark (WMT16 En-De), comparing the LMWT against a baseline self-attention transformer using metrics like BLEU score, perplexity, and token accuracy. Furthermore, we analyze the computational complexity, highlighting the linear scaling of our approach, discuss its novelty in the context of related work, and explore the interpretability offered by visualizing the learned Haar coefficients. Our results indicate that the LMWT achieves competitive performance while offering substantial computational advantages, positioning it as a promising and novel alternative for efficient sequence modeling. 1 Introduction The advent of the transformer architecture [ 14 ] marked a paradigm shift in natural language processing (NLP). Its core innovation, the self-attention mechanism, enables models to weigh the importance of different tokens within an input sequence when computing the representation for each token. This allows for the direct modeling of long-range dependencies, overcoming limitations of previous recurrent and convolutional architectures and leading to groundbreaking performance in tasks like machine translation, text summarization, and question answering. Despite its success, the standard dot-product self-attention mechanism suffers from a critical drawback: its computational and memory complexity scales quadratically, \mathcal{O}(T^{2}) , with the sequence length T . This quadratic scaling poses a significant bottleneck when dealing with long documents, high-resolution images (when adapted for vision), or applications requiring real-time processing. Recognizing this limitation, the research community has actively explored more efficient alternatives. Notable approaches include: • Sparse Attention: Methods like Longformer [ 1 ] and BigBird [ 16 ] use predefined sparsity patterns (e.g., sliding window, global tokens, random patterns) to reduce the number of computed attention scores. • Linearized Attention: Techniques such as Linformer [ 15 ] project the key and value matrices to lower dimensions before the attention computation, achieving linear complexity. Performers [ 2 ] use random feature maps to approximate the softmax kernel, also resulting in linear complexity. • Fourier Transforms: FNet [ 11 ] replaces self-attention entirely with unparameterized Fourier Transforms, mixing tokens with \mathcal{O}(T\log T) complexity. Recent work like SFDLM [ 9 ] utilizes state Fourier diffusion models, demonstrating competitive results without attention’s quadratic cost for long sequences. • Hybrid Approaches: Some methods combine different techniques, like the hybrid Wavelet-Fourier approach for image generation [ 8 ] . • Other Mechanisms: State Space Models (SSMs) and specialized convolutional or recurrent structures have also emerged as alternatives [ 5 , 4 , 3 ] . While these methods offer significant efficiency gains, they often involve approximations, fixed transformations (like standard Fourier transforms), specific sparsity patterns, or different underlying mechanisms that might not optimally capture the complex, hierarchical structures inherent in natural language or other sequential data in the same way as dense attention. In this work, we propose a fundamentally different approach inspired by classical signal processing: the Learnable Multi-Scale Wavelet Transformer (LMWT). We replace the self-attention module with a mechanism based on wavelet transforms, specifically the Haar wavelet transform. Wavelets are renowned for their ability to provide multi-resolution analysis, decomposing signals into components at different scales or frequencies while preserving localization information [ 12 ] . Unlike the Fourier transform, which uses global sinusoidal bases, wavelets employ localized basis functions, making them well-suited for analyzing signals with transient features or non-stationarities, characteristics often found in language. While wavelets have seen use in other deep learning domains, employing a learnable , multi-scale wavelet transform as a direct replacement for the self-attention mechanism within NLP transformers represents a distinct and less explored approach compared to prevalent efficiency techniques based on attention approximation or fixed spectral transforms. The key innovation of our LMWT is twofold: 1. Multi-Scale Decomposition: We utilize the hierarchical nature of the Haar transform to decompose the input sequence representation into approximation (low-frequency, global context) and detail (high-frequency, local features) coefficients across multiple scales. 2. Learnability: Instead of using the fixed Haar basis functions, we introduce learnable parameters within the transform. This allows the model to adapt the wavelet decomposition process end-to-end during training, tailoring the feature extraction to the specific nuances of the data and the requirements of the downstream task. This design offers the potential to capture rich, hierarchical representations efficiently, achieving linear complexity, \mathcal{O}(T) , with respect to sequence length. This paper details the mathematical formulation of the learnable Haar wavelet module (Section 3 ), analyzes its computational complexity compared to self-attention (Section 4 ), describes its integration into the standard transformer architecture (Section 5 ), illustrates the overall architecture (Section 6 ), presents experimental results on a machine translation task (Section 7 ), analyzes the interpretability of learned coefficients (Section 8 ), discusses the implications, novelty, and limitations (Section 9 ), and concludes with future directions (Section 10 ). 2 Background To understand our proposed method, we first briefly review the standard self-attention mechanism and the classical Haar wavelet transform. 2.1 Self-Attention Mechanism In a standard transformer layer [ 14 ] , self-attention operates on an input sequence represented by a matrix \mathbf{X}\in\mathbb{R}^{T\times d} , where T is the sequence length and d is the embedding dimension. The input \mathbf{X} is linearly projected into three matrices: Queries ( \mathbf{Q} ), Keys ( \mathbf{K} ), and Values ( \mathbf{V} ): \mathbf{Q}=\mathbf{X}\mathbf{W}_{Q},\quad\mathbf{K}=\mathbf{X}\mathbf{W}_{K},\quad\mathbf{V}=\mathbf{X}\mathbf{W}_{V}, (1) where \mathbf{W}_{Q},\mathbf{W}_{K}\in\mathbb{R}^{d\times d_{k}} and \mathbf{W}_{V}\in\mathbb{R}^{d\times d_{v}} are learnable weight matrices (often d_{k}=d_{v}=d/h , where h is the number of attention heads). The attention output is computed as: \text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\softmax\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{k}}}\right)\mathbf{V}. (2) The core computation involves the matrix multiplication \mathbf{Q}\mathbf{K}^{\top} , resulting in a T\times T attention score matrix. This step dominates the computation, leading to the \mathcal{O}(T^{2}d_{k}) complexity. Transformers typically employ multi-head attention, where this process is performed in parallel with different projection matrices, and the results are concatenated and linearly projected. 2.2 Classical Haar Wavelet Transform The Haar wavelet transform is the simplest form of wavelet transform [ 6 ] . For a 1D discrete signal x=(x_{0},x_{1},\dots,x_{n-1}) where n is an even number, the first level of the Haar transform computes approximation coefficients ( a ) and detail coefficients ( d ) for pairs of adjacent elements: a_{i}=\frac{x_{2i}+x_{2i+1}}{\sqrt{2}},\quad d_{i}=\frac{x_{2i}-x_{2i+1}}{\sqrt{2}},\quad\text{for }i=0,1,\dots,n/2-1. (3) The approximation coefficients \{a_{i}\} represent a downsampled, smoothed version of the signal, capturing lower-frequency information. The detail coefficients \{d_{i}\} capture higher-frequency information, representing the differences between adjacent pairs. The original signal can be perfectly reconstructed from these coefficients: x_{2i}=\frac{a_{i}+d_{i}}{\sqrt{2}},\quad x_{2i+1}=\frac{a_{i}-d_{i}}{\sqrt{2}}. (4)