On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning

Haoyuan Sun, Ali Jadbabaie, Navid Azizan · 2025

Read On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning by Haoyuan Sun, Ali Jadbabaie, Navid Azizan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18187.txt.

Introduction

Haoyuan Sun, Ali Jadbabaie and Navid Azizan study how feed-forward layers let Transformers learn nonlinear tasks in context, in work from 2025. Transformers can pick up new tasks from a few prompt examples with no weight updates. Prior theory showed plain linear attention mimics one gradient step on linear problems, but the authors prove it cannot beat linear predictors on nonlinear ones.

Adding a gated feed-forward block changes the picture: the block runs one gradient step on a polynomial kernel regression loss. One block is capped by its width, yet stacking blocks spreads richer kernels across depth as block-coordinate descent. Feed-forward layers thus supply the nonlinear power, the authors argue.

Catalog record

CREATORHaoyuan Sun, Ali Jadbabaie, Navid Azizan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18187

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning Haoyuan Sun, Ali Jadbabaie, Navid Azizan Abstract Transformer-based models demonstrate a remarkable ability for in-context learning (ICL), where they can adapt to unseen tasks from a few prompt examples without parameter updates. Notably, recent research has provided insight into how the Transformer architecture can perform ICL, showing that the optimal linear self-attention (LSA) mechanism can implement one step of gradient descent for linear least-squares objectives when trained on random linear regression tasks. Building upon this understanding of linear ICL, we investigate ICL for nonlinear function classes. We first show that LSA is inherently incapable of solving problems that go beyond linear least-squares objectives, underscoring why prior solutions cannot readily extend to nonlinear ICL tasks. To overcome this limitation, we investigate a mechanism combining LSA with feed-forward layers that are inspired by the gated linear units (GLU) commonly found in modern Transformer architectures. We show that this combination empowers the Transformer to perform nonlinear ICL, specifically by implementing one step of gradient descent on a polynomial kernel regression loss. Furthermore, we show that multiple blocks of our GLU-LSA model implement block coordinate descent in this polynomial kernel space. Our findings highlight the distinct roles of attention and feed-forward layers, demonstrating that the feed-forward components provide a mechanism by which Transformers gain nonlinear capabilities for ICL. 0 0 footnotetext: Corresponding author: Haoyuan Sun ( haoyuans [at] mit [dot] edu ). 1 Introduction In recent years, large language models (LLM) based on the Transformer architecture ( Vaswani et al. 2017 ) have achieved remarkable successes across numerous domains such as computer vision ( Dosovitskiy et al. 2021 ) , speech recognition ( Radford et al. 2023 ) , multi-modal data processing ( Yang et al. 2023 ) , and human reasoning tasks ( OpenAI 2023 ) . Beyond these practical achievements, Transformers have also demonstrated the extraordinary ability to perform in-context learning (ICL) ( Brown et al. 2020 ; Wei et al. 2022b ; Min et al. 2022 ) . In ICL, a model adapts to different tasks by leveraging a prompt containing a short sequence of examples without requiring updates to its parameters. The study of in-context learning has attracted significant attention in recent literature. Formally, given a prompt composed of input-label pairs and a query, (x_{1},f(x_{1}),x_{2},f(x_{2}),\dots,x_{n},f(x_{n}),x_{\textsf{query}}) , a Transformer model is said to learn a function class \mathcal{F} in context if it can accurately predict f(x_{\textsf{query}}) for previously unseen functions f\in\mathcal{F} . Empirical studies by Garg et al. 2022 have demonstrated the efficacy of Transformers in performing in-context learning for a range of function classes such as linear functions, decision trees, and ReLU neural networks. Building on these insights, Akyürek et al. 2023 and Von Oswald et al. 2023 established that Transformers can learn linear functions in context by implicitly implementing gradient descent over a linear regression objective on the prompt’s examples and provided a construction that facilitates such a mechanism. Extending this perspective, subsequent theoretical analysis by Ahn et al. 2024b ; Zhang et al. 2024a ; Mahankali et al. 2024 showed that under suitably defined distributions over the prompts, the optimal single-layer linear self-attention (LSA) mechanism can effectively implement one step of gradient descent. Despite these advancements, the theoretical understanding of ICL for nonlinear function classes remains underdeveloped. Prior theoretical explanations primarily focus on the attention layers of Transformers. However, the role of feed-forward layers in ICL tasks, particularly for nonlinear functions, has been underexplored and may be key to a complete understanding. In this work, we demonstrate that the feed-forward layers in Transformers may represent a crucial yet overlooked component to advance the theoretical understanding of ICL for nonlinear functions. As a representative case, we show that using standard feed-forward layer architectures, such as those based on the gated linear units (GLU), enables Transformers to learn quadratic and polynomial functions in context, illustrating a broader significance for feed-forward layers within Transformer models. Our contributions. In this work, we investigate the pivotal role of feed-forward layers in enabling Transformers to learn nonlinear functions in context. Specifically, we show that Transformers with alternating linear attention and feed-forward layers can learn quadratic and higher-order polynomial functions in context by implicitly implementing gradient descent updates with respect to a collection of nonlinear features. Our results overcome the inherent limitations of attention-only mechanisms in nonlinear ICL, identify key properties of optimal layer weights, and present scalable constructions for nonlinear ICL with deeper models. The key contributions of our paper are as follows: \blacksquare We establish the crucial role of feed-forward layers in enabling in-context learning of nonlinear target functions. In Section 3.1 , we show that when the target functions are nonlinear, no deep linear self-attention (LSA) network can achieve lower in-context learning loss than a linear least-squares predictor. Further, in Section 3.2 , we draw inspiration from modern Transformer architectures and show that a Transformer block consisting of one GLU-like feed-forward layer and one LSA layer can implement one step of gradient descent with respect to a quadratic kernel. \blacksquare In Section 3.3 , we analyze the Transformer model featuring the LSA-GLU mechanism. We derive optimal conditions for the weights within both the GLU-like feed-forward layer and the attention layer. In particular, we show that the optimal choice of the feed-forward layer must correspond to a kernel function that computes all possible quadratic features. Furthermore, we show that the optimal attention layer implements gradient descent on the kernel feature space with a preconditioner approximately equal to the negative inverse of the features’ covariance matrix. \blacksquare In Section 4 , we describe several challenges with the initial construction in Section 3.2 and propose a deeper Transformer model to overcome them, yielding improved results for learning quadratic functions in-context. Furthermore, in Section 4.2 , we highlight that the same approach can be easily generalized to handle more complex nonlinear tasks, including learning higher-order polynomial functions in context, by distributing the computation of kernel functions across multiple layers. We support our analysis throughout this section with extensive numerical experiments. Related works. This section highlights the most directly related works in the literature; a more detailed survey of the literature can be found in Appendix A . While prior work has primarily focused on the roles of attention layers in in-context learning, the significance of feed-forward layers has been largely overlooked. For instance, Zhang et al. 2024b showed that incorporating an MLP feed-forward layer allows for implementing gradient descent with learnable nonzero initialization. However, the results of this work are still limited to linear ICL. To the best of our knowledge, Zhang et al. 2024b provides the only prior theoretical analysis exploring a potential mechanism by which Transformer feed-forward layers contribute to ICL. While the theoretical analysis of in-context learning for nonlinear function classes has gathered significant attention in the literature, the existing works remain limited. First, Cheng et al. 2024 shows that attention endowed with certain nonlinear activation functions can implement gradient descent on a function space induced by this nonlinearity. While this result allows for learning a range of nonlinear functions, it requires a non-standard form of attention. Another line of work ( Huang et al. 2024 ; Yang et al. 2024b )

Read the complete text