Reversed Attention: On The Gradient Descent Of Attention Layers In GPT
Shahar Katz Lior Wolf · 2024
Read Reversed Attention: On The Gradient Descent Of Attention Layers In GPT by Shahar Katz Lior Wolf (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17019.txt.
Introduction
Shahar Katz and Lior Wolf studied in 2024 the backward pass of attention in GPT-style language models, a direction most research ignores. Attention is usually analyzed through the values computed while the model reads text. Training, though, sends gradients backward through the same layers. The authors show this backward flow implicitly builds a second attention matrix, which they call reversed attention.
They examine its properties and show it can reveal how the model behaves and learns. With a technique they name attention patching, reversed attention can directly edit the forward pass without changing any weights. The work makes the training side of attention more interpretable. It also offers a new handle for steering model behavior.
Catalog record
| CREATOR | Shahar Katz Lior Wolf |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17019
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reversed Attention: On The Gradient Descent Of Attention Layers In GPT Shahar Katz Lior Wolf Abstract The success of Transformer-based Language Models (LMs) stems from their attention mechanism. While this mechanism has been extensively studied in explainability research, particularly through the attention values obtained during the forward pass of LMs, the backward pass of attention has been largely overlooked. In this work, we study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as “Reversed Attention”. We examine the properties of Reversed Attention and demonstrate its ability to elucidate the models’ behavior and edit dynamics. In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model’s weights, using a novel method called “attention patching”. In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass. Our code will be available at: https://github.com/shacharKZ/Reversed-Attention . Figure 1: In this paper we examine the attention maps obtained from the backward pass, which we named “Reversed Attention” (RA). This example present the forward and backward pass of a single attention head of GPT2-xl when prompt with “I like Italy and France, I visited the city of”. After the model answer “Florence”, a city in Italy, we apply a backward pass with “Florence” as the target for the loss and produce the RA maps. Between all the 1200 attention heads this model has, the presented head has the highest RA map’s norm. Compared to the forward attention map, the RA map is more sparse and interpretable. This RA demonstrates how the backpropagation attempts to amplify the information from the token “Italy” (red) while reducing the influence of “France” (blue). 1 Introduction The widespread use of automatic gradient technologies such as AutoGrad ( Maclaurin, 2016 ; Paszke et al., 2019 ) to obtain the gradients may cause the explicit derivations of these gradients to be overlooked. In this work we explore the equations that govern the backpropagation of the popular Transformer-based ( Vaswani et al., 2017 ) GPT ( Radford et al., 2018 ) architecture. This explicit analysis leads to several surprising discoveries. First, while during the forward pass the attention mechanism explicitly creates triangular attention matrices, from multiplying the query and key vectors, during the backward pass it implicitly creates triangular matrices that determine the gradients of the queries and keys. Due the similarity of these triangular matrices to the forward attention, we refer to them as the Reversed Attention (RA) matrices, Figure 1 . Secondly, we study the effect induced by GD and how it tries to increase or decrease attention scores between the queries and keys of each attention head. Based on these discoveries, we explore the use of RA in LM explainability. While the forward pass attention, which fails in providing a clear explanation of the model’s behavior ( Jain and Wallace, 2019 ) , we demonstrate RA’s ability to achieve competitive results in concert with methods such as causal mediation ( Vig et al., 2020 ; Meng et al., 2022 ) in perturbation analysis. Furthermore, one can view the RA as a correction term to the attention, given the loss of the backward pass. As an application, we inject the RA scores directly into the forward pass of attention, in order to modify the model’s predictions. This novel method, which we call “attention patching”, does not involve any parameter updates and offers a new perspective on how interventions can be performed on LMs. Our main contributions are as follows: (i) We provide a mathematical walk-through and interpretation of the gradients and Vector-Jacobian Products (VJPs) governing the backpropagation of GPT. (ii) We identify the attention-softmax derivative as an implicit attention map, which we term Reversed Attention (RA). (iii) We visualize and qualitatively explore the interpretability of RA. (iv) We conduct a perturbation test to quantify the explainability of RA. (v) We demonstrate a novel patching method that uses RA to edit LM’s predictions. 2 Related Work All leading deep learning models are trained using variants of Gradient Descent (GD), an implementation of the backpropagation algorithm. While much research examines the impact of GD on GPTs’ performance, the internal computations of this process often remain a black box ( Radford et al., 2018 ; Radford et al., 2019 ; Gururangan et al., 2020 ) . Some studies simplify the Transformer architecture to understand GD, for instance, reducing multi-head attention to a single head or linear attention in toy models ( Tian et al., 2023 ; Tarzanagh et al., 2023 ; Mahankali et al., 2023 ; Dai et al., 2023 ) . Notably, the literature on the backward pass of full multi-head attention is limited. Our work addresses this gap by examining assumption-free full GPT models, with a specific emphasis on detailing the mathematical computation of gradients. Previous investigations into weight updates via GD have focused primarily on the MLP layers or the data it was trained on ( Gueta et al., 2023 ) . Recently, Katz et al. (2024) has revealed that gradients can be interpreted as tokens’ embeddings. Specifically, the Vector Jacobian Products (VJPs) that are passed by the residual from the last layer’s loss to earlier layers can be seen as an intermediate state toward the model as it adjusts its weights. Gradients have been leveraged to localize the source of specific model behavior for a given input ( Simonyan et al., 2014 ; Ancona et al., 2018 ) or specific components of the models ( Barkan et al., 2021 ; Ma et al., 2023 ) . In our study, we explore the use of RA as a means to quantify the relative influence of the various components of a model. We compare our method with the current leading technique for such localization, Causal Mediation (CM) ( Vig et al., 2020 ; Meng et al., 2022 ; Meng et al., 2023 ) . CM involves probing the effect of altering components during the forward pass, which necessitates significantly more computation compared to the RA approach. Recent works explore the patching technique, where one model’s intermediate state is integrated into the forward pass of another ( Zhang and Nanda, 2023 ; Elhage et al., 2021 ; Wang et al., 2023 ; Todd et al., 2023 ) . While they focus on activation patching, we propose a novel approach: directly injecting attention scoring maps of each attention head, without parameter updates. As far as we can ascertain, we are the first to identify, visualize, and explore the dynamics of the backward pass using RA maps. 3 Background This section provides the necessary background and establishes the notation used. Generative Pre-trained Transformer (GPT) is an auto-regressive architecture of multiple transformer blocks connected via a residual stream. As input, a GPT model receives a sequence of n tokens (a prompt) and predicts a single token. An embedding matrix at the start of the model embeds the token into vectors X=[{x^{1},\cdots,x^{n}}]\in\mathbb{R}^{n\times d} where d is an embedding dimension. At the end of the model, the embedded predictions are projected back into tokens using a decoding matrix. Each transformer block consists of