Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

Haoran Zhang, Feng Zhou · 2026

Read Flexformer: Flexible Linear Transformer with Learnable Attention Kernel by Haoran Zhang, Feng Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27748.txt.

Introduction

Transformers pay a quadratic cost for attention on long sequences. Kernel-based linear attention cuts that cost. Its kernels are usually fixed or only weakly learnable, which limits what the model can express. Haoran Zhang and Feng Zhou propose Flexformer in 2026. Flexformer learns its kernels from data in a fully driven way. It builds on random Fourier features. The spectral frequencies become trainable parameters.

Learning the frequencies lets the model cover a broad family of kernels. Two variants are developed. One is stationary, and the other is nonstationary with strictly greater expressiveness. Tests cover language modelling and sequence classification. Flexformer consistently outperformed the baselines there. A second strength is transfer. It can be distilled from a pretrained Transformer and still recover softmax attention. Its learned kernels also move well across domains. The authors report high efficiency together with strong results on long-sequence tasks.

Catalog record

CREATORHaoran Zhang, Feng Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27748

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel Haoran Zhang, Feng Zhou Abstract Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences. Kernel-based linear attention reduces this complexity but typically relies on fixed or weakly learnable kernels, restricting expressiveness and performance. In this work, we propose Flexformer, a flexible linear Transformer that learns attention kernels in a fully data-driven manner. Flexformer builds on random Fourier feature-based linear attention and treats spectral frequencies as trainable parameters, enabling the model to learn a broad family of attention kernels. We develop both stationary and nonstationary variants, with the latter offering strictly greater expressiveness. Extensive experiments on language modeling and sequence classification demonstrate that Flexformer consistently outperforms baselines. Moreover, Flexformer can be effectively distilled from pretrained Transformers to recover softmax attention and exhibits strong kernel transferability across domains, achieving both high efficiency and competitive performance on long-sequence tasks. 1 Introduction The Transformer 38 architecture has achieved remarkable success in machine translation tasks since its original proposal. Subsequently, numerous variants of Transformer have been extensively applied across prominent domains within artificial intelligence, including natural language processing 9 ; 18 ; 5 , computer vision 10 ; 20 , time series analysis 43 ; 25 ; 19 , and audio processing 33 ; 11 . The most critical component in these Transformer-based models is the dot-product attention mechanism. It operates by computing pairwise token affinities across the entire input sequence, thereby enabling each token to dynamically aggregate the most relevant information pertaining to itself, while effectively modeling long-range dependencies within the sequence. Unlike sequential RNNs 34 processing tokens step-by-step, Transformer attention computes representations for all positions in parallel, maximizing computational hardware utilization. This effective attention mechanism, however, suffers from efficiency limitations: it computes attention weights for all pairs of positions in the sequence, leading to both time and space complexity that grow quadratically with the sequence length. To address this issue, numerous studies have attempted to optimize the computation of the attention mechanism. These redesigned or optimized attention computation methods should effectively reduce time and space complexity while achieving performance that comparable or even surpasses that of softmax attention. Some works are based on sparse attention 15 ; 32 ; 41 , which modifies the attention patterns by restricting the attention range of each token to focus only on a subset of critical tokens. However, these attention patterns are not rigorously validated and are often based on empirically derived patterns, making it difficult to guarantee their consistently superior performance across diverse data domains. The other category is kernel-based linear attention 13 ; 7 ; 29 , which represents the process of computing dot products and applying softmax using kernels. The kernel can be written as an inner product of feature maps. By rearranging the order of matrix multiplications, the quadratic complexity of attention can be reduced to linear complexity. Figure 1 illustrates the difference of the computational processes between kernel-based linear attention and softmax attention. (a) Softmax attention. (b) Kernel-based linear attention. Figure 1 : Comparison between softmax attention and kernel-based linear attention. Softmax attention requires \mathcal{O}(N^{2}d) time complexity and \mathcal{O}(N^{2}+Nd) space complexity, whereas kernel-based linear attention achieves \mathcal{O}(Ndd^{\prime}) time complexity and \mathcal{O}(Nd+Nd^{\prime}+dd^{\prime}) space complexity, where N,d,d^{\prime} denote input sequence length, hidden dimension, and feature map dimension respectively. When N\gg d and N\gg d^{\prime} , linear attention achieves significant savings in both time and space. Here, we focus on the kernel-based linear attention whose core lies in constructing an effective feature map. While numerous methods aim to recover softmax attention, there is still no proof that softmax is invariably optimal under all circumstances. Therefore, a more effective approach is to apply learnable kernels that are directly learned from data. However, how to construct sufficiently flexible learnable kernels that guarantee strong performance remains an open problem. Hedgehog 42 and Polaformer 22 empirically exploit the low-entropy property of softmax attention by combining learnable mappings with spiky functions such as exponentials and power functions. However, these methods do not guarantee that the learned kernel is sufficiently expressive to encompass a wide variety of kernels, which can compromise their performance. In this work, we propose a Flex ible Linear Trans former with learnable kernel ( Flexformer ). Flexformer extends kernel-based linear attention with random Fourier features by making the feature map fully learnable. In contrast to existing learnable kernel methods, Flexformer is able to model a substantially broader and more expressive family of kernels. Specifically, spectral representation theory states that a kernel admits an inverse Fourier transform representation in terms of its spectral measure, which can be approximated through Monte Carlo sampling. In Flexformer, instead of sampling frequencies from a fixed spectral density, we treat the frequencies as learnable parameters and optimize them directly from data, thereby enabling the attention kernel to be learned in a data-driven manner. We propose both stationary and nonstationary versions of Flexformer, with the nonstationary formulation being strictly more expressive. Theoretically, we can show that the kernel family of Flexformer contains the softmax kernel. This guarantees that Flexformer can, at a minimum, learn a softmax attention, and potentially learn attention patterns that outperform softmax. Flexformer can be trained either from scratch or by distilling 42 from a pretrained Transformer. We conduct extensive experiments on multiple benchmarks, including language modeling and classification tasks, demonstrating Flexformer’s superior performance over existing baselines, as well as its strong capability to recover the softmax kernel and effectively transfer the learned kernel. As a variant of linear attention, it significantly improves efficiency and reduces memory consumption when processing extremely long sequences. Specifically, our contributions are summarized as follows: • We propose Flexformer, which leverages learnable attention kernels constructed via random Fourier features, achieving high expressiveness while preserving linear time and space complexity w.r.t. sequence length. • We evaluate Flexformer through from-scratch training on language modeling and sequence classification tasks. The results show that Flexformer consistently outperforms other linear-attention baselines, highlighting its strong effectiveness and efficiency. • Flexformer can faithfully approximate softmax attention through attention weight distillation, enabling efficient linear-attention replacements with preserved performance, and demonstrates strong kernel transferability in cross-domain settings. 2 Preliminaries In this section, we introduce the background knowledge of self-attention mechanism and kernel-based linear attention. 2.1 Self-attention Mechanism Given an input sequence \mathbf{X}=[\text{X}_{1},\cdots,\text{X}_{N}]^{\top}\in\mathbb{R}^{N\times d} , where N and d denote the input length and embedding dimension, respectively. Each token \text{X}_{i} is projected through linear maps to generate the query, key, and value tensors, formally expressed as \text{Q}_{i}=\text{W}_{Q}\text{X}_{i},\text{K}_{i}=\text{W}_{K}\text{X}_{i},\text{V}_{i}=\text{W}_{V}\text{X}_{i},i=1,\cdots,N . For each query tensor, a similarity measure is computed with all key tensors. These similarity scores are then normalized to form attention weights, which are used to perform a weighted summation over the corresponding value tensors, thereby generating context-aware representations. Formally, the output of the self-attention module for the i -th token can be expressed as: \text{O}_{i}=\frac{\sum\limits_{j=1}^{N}\text{sim}(\text{Q}_{i},\text{K}_{j})\text{V}_{j}}{\sum\limits_{j=1}^{N}\text{sim}(\text{Q}_{i},\text{K}_{j})}, (1) where \text{sim}(\cdot,\cdot):\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}_{+} denotes the similarity measure function. In vanilla Transformer 38 with softmax attention, \text{sim}(\text{Q}_{i},\text{K}_{j}) is defined as \text{sim}(\text{Q}_{i},\text{K}_{j})=\exp\left(\frac{\text{Q}_{i}^{\top}\text{K}_{j}}{\sqrt{d}}\right) . 2.2 Kernel-based Linear Attention If we directly follow the computation order in Equation 1 by first calculating the similarity between queries and keys at all pairwise positions to obtain a N\times N attention weight matrix, and then multiplying with values,

Read the complete text