Beyond Linearity in Attention Projections: The Case for Nonlinear Queries

Marko Karbevski · 2026

Read Beyond Linearity in Attention Projections: The Case for Nonlinear Queries by Marko Karbevski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13381.txt.

Introduction

Marko Karbevski argues for nonlinear queries in transformer attention projections. Recent algebraic work shows the Query projection can be set to identity with little loss of skill. Attention depends on the input only through set products, so basis shifts are absorbed by nearby layers. Karbevski swaps the linear Query map for a nonlinear residual with a bottleneck MLP. The identity term anchors the new map to a known-good prior. On GPT-3 small style models, the change cut validation loss and perplexity, beating a larger baseline.

Catalog record

CREATORMarko Karbevski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17540
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Linearity in Attention Projections: The Case for Nonlinear Queries Thanks: To appear at the ICLR 2026 Workshop on Geometry-grounded Representation Learning and Generative Modeling (GRaM) . Code and checkpoints: https://github.com/MarkoKarbevski/beyond_query_linearity Marko Karbevski Abstract Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection W Q W_{Q} may be set to identity without noticeable performance deterioration. This is possible because attention depends on X X only through the products X ​ W Q , X ​ W K , X ​ W V XW_{Q},XW_{K},XW_{V} , allowing basis transformations to be absorbed by adjacent layers and propagated through the network. We replace W Q ∈ ℝ d × d W_{Q}\in\mathbb{R}^{d\times d} with a nonlinear residual of the form Q ⁡ ( X ) = X + f θ ​ ( X ) Q(X)=X+f_{\theta}(X) , where f θ f_{\theta} is a bottleneck MLP with d 2 + O ⁡ ( d ) d^{2}+O(d) parameters. The identity term anchors the nonlinearity to a known-good prior. Experiments on GPT-3 small style models show consistent improvement over the baseline, comfortably outperforming a model with 12.5% more non-embedding parameters. These results motivate investigation at larger scales and across modalities. 1 Introduction The transformer architecture [ 22 ] computes attention via Query, Key, and Value projections, each parameterized by learned weight matrices W_{Q},W_{K},W_{V}\in\mathbb{R}^{d\times d} . Recent work has revealed a fundamental invariance in the attention computation that renders the Query projection algebraically redundant. This observation, grounded in the geometry of the transformer computation graph, motivates a principled modification: replacing the redundant linear projection with a nonlinear one. Algebraic and topological structure. Consider self-attention transformers with arbitrary masking patterns, which subsumes encoder-only architectures, decoder-only architectures, and certain cross-attention configurations such as prefix language models where the prefix serves as encoder context. Let multi-head attention be defined as \text{MHA}(X)=\text{Concat}(\text{head}_{1},\ldots,\text{head}_{h})W_{O} , where each head computes \text{head}_{i}=\text{Attention}(XW_{Q}^{i},XW_{K}^{i},XW_{V}^{i}) with W_{Q}^{i},W_{K}^{i},W_{V}^{i} the per-head projection matrices. Writing W_{Q}=(W_{Q}^{1}|\cdots|W_{Q}^{h}) and similarly for W_{K},W_{V} , attention depends on X only through the products XW_{Q} , XW_{K} , XW_{V} . Karbevski and Mijoski [ 12 ] establish that this admits a reparametrization invariance: for any invertible \Theta , (X,W_{Q},W_{K},W_{V},W_{O})\mapsto(X\Theta,\Theta^{-1}W_{Q},\Theta^{-1}W_{K},\Theta^{-1}W_{V},W_{O}) (1) leaves the MHA output unchanged. Since singular matrices have Lebesgue measure zero, W_{Q} , W_{K} , and W_{V} are almost surely invertible, allowing the choice \Theta=W_{Q} (or W_{K} , W_{V} ) to simplify one of the three matrices. Under mild conditions on the network topology, this basis change propagates through the entire network: each layer passes its transformation to its predecessor, telescoping back to the embedding layer. Graef [ 4 ] first proved this propagation for simplified architectures; Karbevski and Mijoski [ 12 ] extended it to standard transformers with layer normalization and residual connections, verifying W_{Q}=I empirically. Setting \Theta=W_{Q} yields W_{Q}\mapsto I , establishing that W_{Q} is algebraically redundant. Crucially, this extends to Grouped Query Attention: since GQA shares W_{K} and W_{V} across query groups, only W_{Q} can be eliminated without disrupting the shared structure, making queries the natural choice. The invariance is also compatible with Mixture-of-Experts. While we test with learned positional embeddings, this reparametrization also holds for RoPE [ 21 ] , as shown by Karbevski and Mijoski [ 12 ] . Identity as a good prior. Empirically, models trained with W_{Q}=I match baseline performance while remaining stable at 3\times lower weight decay [ 12 ] . The identity is thus a good prior for queries, both for representability and for training stability. Proposal. The work of Graef [ 4 ] and Karbevski and Mijoski [ 12 ] establishes that any linear expenditure of parameters at W_{Q} is largely redundant: the learned matrix can be absorbed into adjacent layers without changing the function computed. If we wish to allocate parameters directly to the query pathway, nonlinearity is therefore imposed. Following the suggestion of Karbevski and Mijoski [ 12 ] , we implement a nonlinear query projection directly: Q(X)=(X+f_{\theta}(X))/2 (2) where f_{\theta} is a bottleneck MLP with d^{2}+O(d) learnable parameters, the same order as the W_{Q} it replaces. The 1/2 scaling follows Karbevski and Mijoski [ 12 ] . The identity term X anchors the nonlinearity to a known-good prior while providing a direct path for gradient flow [ 5 ] , preserving the stability established for W_{Q}=I . Keys and values remain standard linear projections. Specifically, we implement f_{\theta} as: f_{\theta}(X)=\text{LN}\left(\text{GELU}\left(\text{RMSNorm}(X)W_{1}\right)W_{2}\right) (3) where W_{1}\in\mathbb{R}^{d\times r} , W_{2}\in\mathbb{R}^{r\times d} , and r=d/2 . The query Q(X)=(X+f_{\theta}(X))/2 is applied independently per token. The attention logits Q^{i}{K^{i}}^{\top}/\sqrt{d_{k}} per head use standard scaling. Matrix parameters total 2dr=d^{2} ; normalization layers add O(d) parameters ( <0.1\% overhead). The design prioritizes stability over throughput or final quality; we conjecture that this formulation may be overly conservative. The choice of LN and RMSNorm was made for stability in initial experiments; systematic exploration of normalization variants, or their elimination entirely, is an immediate priority for future work. Related work. Kernel-based attention methods [ 2 , 14 ] apply nonlinear feature maps \phi(Q),\psi(K) to achieve linear complexity, but retain standard linear projections Q=XW_{Q} before the nonlinearity. MLP-Attention [ 18 ] eliminates Q and K projections entirely, replacing the QK^{\top} computation with an MLP that maps embeddings directly to attention weights; however, this adds substantial parameters ( {\sim}10\% ), was validated only at small scale ( {\sim}5 M parameters on character-level Tiny Shakespeare), uses no residual structure, and lacks theoretical motivation. Zhang [ 24 ] replaces all of Q, K, V with two-layer feedforward MLPs (no residual), also adding substantial parameters; they fine-tune pretrained models rather than training from scratch, and offer no algebraic justification. Nonlinear extensions of LoRA [ 16 , 9 , 3 ] introduce nonlinearities to break the low-rank bottleneck of adapters in parameter-efficient fine-tuning; this is a distinct setting from base architecture design, with different goals and justification. To our knowledge, this is the first direct implementation of nonlinear query projections as suggested by Karbevski and Mijoski [ 12 ] : a principled, parameter-neutral architectural modification with residual structure for pretraining, where nonlinearity is introduced because there is strong empirical and theoretical evidence that learned linearity is redundant. 2 Motivation 2.1 Linear Projections and the Decoding Bottleneck Let X\in\mathbb{R}^{n\times d} denote the input to an attention layer, where n is the sequence length and d is the model dimension. Write x\in\mathbb{R}^{1\times d} for a single token. The input to each attention layer is the output of the previous block: x=x_{\text{prev}}+\text{MLP}(x_{\text{prev}}) . From this single x , standard attention generates four vectors in \mathbb{R}^{1\times d} : query q=xW_{Q} , key k=xW_{K} , value v=xW_{V} , and residual x . All four are linear functions of x . This creates a bottleneck: the four outputs cannot vary independently. Making the query nonlinear, q=x+f_{\theta}(x) , partially decouples it. The query gains independent variation, leaving more freedom in how x serves keys, values, and the residual. This bottleneck may be more pronounced in Mixture-of-Experts architectures, where experts typically operate with reduced latent dimension; we leave this investigation to future work. 2.2 Evidence for Functional Specialization The projections W_{Q} , W_{K} , W_{V} are not interchangeable. Within each head, if W_{Q}^{i}=W_{K}^{i} , then Q^{i}{K^{i}}^{\top}=XW_{Q}^{i}{W_{Q}^{i}}^{\top}X^{\top} is symmetric; empirically, trained attention matrices are not symmetric. Hu et al. [ 8 ] report that low-rank

Read the complete text