Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
Yingcong Li ⋆, Davoud Ataee Tarzanagh ⋆, Ankit Singh Rawat, Maryam Fazel, Samet Oymak · 2025
Read Gating is Weighting: Understanding Gated Linear Attention through In-context Learning by Yingcong Li ⋆, Davoud Ataee Tarzanagh ⋆, Ankit Singh Rawat, Maryam Fazel, Samet Oymak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04308.txt.
Introduction
Yingcong Li and co-authors wrote this machine learning theory paper, published in two thousand twenty-five, on what gated linear attention really does. Such designs, found in models like Mamba and RWKV, promise the speed of simple attention with extra control gates.
The team studies the design through in-context learning, where a model learns from its prompt. They show a stack of gated layers can carry out a broad family of weighted gradient descent methods. The gates, shaped by the input, act as the weights, deciding how much each token counts. A new data model with multitask prompts lets them map the learning landscape. Under mild assumptions, that landscape has a single best solution, up to scaling. The results explain when gating beats plain linear attention and why.
Catalog record
| CREATOR | Yingcong Li ⋆, Davoud Ataee Tarzanagh ⋆, Ankit Singh Rawat, Maryam Fazel, Samet Oymak |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 118 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-04308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gating is Weighting: Understanding Gated Linear Attention through In-context Learning Yingcong Li ⋆, Davoud Ataee Tarzanagh ⋆, Ankit Singh Rawat, Maryam Fazel, Samet Oymak Abstract Linear attention methods offer a compelling alternative to softmax attention due to their efficiency in recurrent decoding. Recent research has focused on enhancing standard linear attention by incorporating gating while retaining its computational benefits. Such Gated Linear Attention (GLA) architectures include competitive models such as Mamba and RWKV. In this work, we investigate the in-context learning capabilities of the GLA model and make the following contributions. We show that a multilayer GLA can implement a general class of Weighted Preconditioned Gradient Descent (WPGD) algorithms with data-dependent weights. These weights are induced by the gating mechanism and the input, enabling the model to control the contribution of individual tokens to prediction. To further understand the mechanics of this weighting, we introduce a novel data model with multitask prompts and characterize the optimization landscape of learning a WPGD algorithm. Under mild conditions, we establish the existence and uniqueness (up to scaling) of a global minimum, corresponding to a unique WPGD solution. Finally, we translate these findings to explore the optimization landscape of GLA and shed light on how gating facilitates context-aware learning and when it is provably better than vanilla linear attention. † † footnotetext: 1 University of Michigan, {yingcong,oymak}@umich.edu . 2 University of Pennsylvania, [email protected] . 3 Google Research NYC, [email protected] . 4 University of Washington, [email protected] . ⋆ Equal contribution. 1 Introduction The Transformer ( Vaswani, 2017 ) has become the de facto standard for language modeling tasks. The key component of the Transformer is the self-attention mechanism, which computes softmax-based similarities between all token pairs. Despite its success, the self-attention mechanism has quadratic complexity with respect to sequence length, making it computationally expensive for long sequences. To address this issue, a growing body of work has proposed near-linear time approaches to sequence modeling. The initial approaches included linear attention and state-space models, both achieving O(1) inference complexity per generated token, thanks to their recurrent form. While these initial architectures typically do not match softmax attention in performance, recent recurrent models such as Mamba ( Gu & Dao, 2023 ) , mLSTM ( Beck et al., 2024 ) , GLA Transformer ( Yang et al., 2023 ) , and RWKV-6 ( Peng et al., 2024 ) achieve highly competitive results with the softmax Transformer. Notably, as highlighted in Yang et al. (2023) , these architectures can be viewed as variants of gated linear attention (GLA), which incorporates a gating mechanism within the recurrence of linear attention. Given a sequence of tokens ({\bm{z}}_{i})_{i=1}^{n+1}\in\mathbb{R}^{d+1} and the associated query, key, and value embeddings ({\bm{q}}_{i},\bm{k}_{i},\bm{v}_{i})_{i=1}^{n+1}\subset\mathbb{R}^{d+1} , with d+1 being the embedding dimension, the GLA recurrence is given by \begin{split}&{{\bm{S}}}_{i}={\bm{G}}_{i}\odot{{\bm{S}}}_{i-1}+\bm{v}_{i}\bm{k}_{i}^{\top},\quad\text{and}\\ &\bm{o}_{i}={{\bm{S}}}_{i}{\bm{q}}_{i},\quad i\in\left\{1,\ldots,n+1\right\}.\end{split} (GLA) Here, {{\bm{S}}}_{i}\in\mathbb{R}^{(d+1)\times(d+1)} represents the 2D state variable with {{\bm{S}}}_{0}=\mathbf{0} ; \bm{o}_{i}\in\mathbb{R}^{d+1} represents the i th output token 1 1 1 The sequence length and embedding dimension are set to n+1 and d+1 per the prompt definition in ( 3 ). ; and the gating variable {\bm{G}}_{i}:=G({\bm{z}}_{i})\in\mathbb{R}^{(d+1)\times(d+1)} is applied to the state {{\bm{S}}}_{i-1} through the Hadamard product \odot and G represents the gating function. When {\bm{G}}_{i} is a matrix of all ones, ( GLA ) reduces to causal linear attention ( Katharopoulos et al., 2020 ) . The central objective of this work is to enhance the mathematical understanding of the GLA mechanism. In-context learning (ICL), one of the most remarkable features of modern sequence models, provides a powerful framework to achieve this aim. ICL refers to the ability of a sequence model to implicitly infer functional relationships from the demonstrations provided in its context window ( Brown, 2020 ; Min et al., 2022 ) . It is inherently related to the model’s ability to emulate learning algorithms. Notably, ICL has been a major topic of empirical and theoretical interest in recent years. More specifically, a series of works have examined the approximation and optimization characteristics of linear attention, and have provably connected linear attention to the preconditioned gradient descent algorithm ( Von Oswald et al., 2023 ; Ahn et al., 2024 ; Zhang et al., 2024 ) . Given that the GLA recurrence in ( GLA ) has a richer design space, this leads us to ask: What learning algorithm does GLA emulate in ICL? How does its optimization landscape behave in multi-task prompting? Contributions: The GLA recurrence in ( GLA ) enables the sequence model to weight past information in a data-dependent manner through the gating mechanism ({\bm{G}}_{i})_{i=1}^{n} . Building on this observation, we demonstrate that a GLA model can implement a data-dependent Weighted Preconditioned Gradient Descent (WPGD) algorithm. Specifically, a one-step version of this algorithm with scalar gating, where all entries of {\bm{G}}_{i} are identical, is described by the prediction: \displaystyle\hat{y}=\bm{x}^{\top}{\bm{P}}{\bm{X}}^{\top}(\bm{y}\odot\bm{\omega}). (1) Here, {\bm{X}}\in\mathbb{R}^{n\times d} is the input feature matrix; \bm{y}\in\mathbb{R}^{n} is the associated label vector; \bm{x}\in\mathbb{R}^{d} represents the test/query input to predict; {\bm{P}}\in\mathbb{R}^{d\times d} is the preconditioning matrix; and \bm{\omega}\in\mathbb{R}^{n} weights the individual samples. When \bm{\omega} is fixed, we drop “data-dependent” and simply refer to this algorithm as the WPGD algorithm. However, for GLA, the weighting vector \bm{\omega} depends on the data through recursive multiplication of the gating variables. Building on this formalism, we make the following specific contributions: \diamond ICL capabilities of GLA (§ 3 ): Through constructive arguments, we demonstrate that a multilayer GLA model can implement data-dependent WPGD iterations, with weights induced by the gating function. This construction sheds light on the role of causal masking and the expressivity distinctions between scalar- and vector-valued gating functions. \diamond Landscape of one-step WPGD (§ 4 ): The GLA \Leftrightarrow WPGD connection motivates us to ask: How does WPGD weigh demonstrations in terms of their relevance to the query? To address this, we study the fundamental problem of learning an optimal WPGD algorithm: Given a tuple ({\bm{X}},\bm{y},\bm{x},y)\sim{\cal{D}} , with y\in\mathbb{R} being the label associated with the query, we investigate the population risk minimization: \begin{split}&{\cal{L}}_{\texttt{WPGD}}^{\star}:=\min_{{\bm{P}}{\in\mathbb{R}^{d\times d}},\bm{\omega}\in\mathbb{R}^{n}}~{\cal{L}}_{\texttt{WPGD}}({\bm{P}},\bm{\omega}),~\\ &\text{where}~~{\cal{L}}_{\texttt{WPGD}}({\bm{P}},\bm{\omega})={\mathbb{E}}_{{\cal{D}}}\left[\left(y-\bm{x}^{\top}{\bm{P}}{\bm{X}}^{\top}(\bm{y}\odot\bm{\omega})\right)^{2}\right].\end{split} (2) As our primary mathematical contribution, we characterize the loss landscape under a general multitask data setting, where the tasks associated with the demonstrations ({\bm{X}},\bm{y}) have varying degrees of correlation to the target task (\bm{x},y) . We carefully analyze this loss landscape and show that, under mild conditions, there is a unique global minimum ({\bm{P}},\bm{\omega}) up to scaling invariance, and the associated WPGD algorithm is also unique. \diamond Loss landscape of one-layer GLA (§ 5 ): The landscape is highly intricate due to the recursively multiplied gating variables. We show that learning the optimal GLA layer can be connected to solving ( 2 ) with a constraint \bm{\omega}\in{\cal{W}} , where the restriction {\cal{W}} is induced by the choice of gating function and input space. Solidifying this connection, we introduce a multitask prompt model under which we characterize the loss landscape of GLA and the influence of task correlations. Our analysis and experiments reveal insightful distinctions between linear attention, GLA with scalar gating, and GLA with vector-valued gating. 2 Problem setup Notations. \mathbb{R}^{d} is the d -dimensional real space, with \mathbb{R}^{d}_{+} and \mathbb{R}^{d}_{++} as its positive and strictly positive orthants. The set [n] denotes \{1,\dots,n\} . Bold letters, e.g., \bm{a} and {\bm{A}} , represent vectors and matrices. The identity