Inferring Functionality of Attention Heads from their Parameters

Amit Elhelo Mor Geva · 2024

Read Inferring Functionality of Attention Heads from their Parameters by Amit Elhelo Mor Geva (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11965.txt.

Introduction

Amit Elhelo and Mor Geva read the job of each attention head straight from its weights, in a 2024 paper. Attention heads are the moving parts of large language models, and past work probed them by watching them run on chosen tasks. That view is slow and partial. The team's frame, MAPS, skips running the model at all and infers each head's role from its stored numbers alone.

Tested on 20 named jobs across six known models, the guesses matched what heads did when run, and cutting heads by the map changed outputs as foretold. The maps also surfaced heads doing jobs that past studies had missed. Some jobs appear in every model, while some quirks trace to each model's build. An auto pipeline then sums up each head's main job in words that human judges found sound for most heads.

Catalog record

CREATORAmit Elhelo Mor Geva
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11965

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inferring Functionality of Attention Heads from their Parameters Amit Elhelo Mor Geva Abstract Attention heads are one of the building blocks of large language models (LLMs). Prior work on investigating their operation mostly focused on analyzing their behavior during inference for specific circuits or tasks. In this work, we seek a comprehensive mapping of the operations they implement in a model. We propose MAPS (Mapping Attention head ParameterS), an efficient framework that infers the functionality of attention heads from their parameters, without any model training or inference. We showcase the utility of MAPS for answering two types of questions: (a) given a predefined operation, mapping how strongly heads across the model implement it, and (b) given an attention head, inferring its salient functionality. Evaluating MAPS on 20 operations across 6 popular LLMs shows its estimations correlate with the head’s outputs during inference and are causally linked to the model’s predictions. Moreover, its mappings reveal attention heads of certain operations that were overlooked in previous studies, and valuable insights on function universality and architecture biases in LLMs. Next, we present an automatic pipeline and analysis that leverage MAPS to characterize the salient operations of a given head. Our pipeline produces plausible operation descriptions for most heads, as assessed by human judgment, while revealing diverse operations. We release our code and mappings at https://github.com/amitelhelo/MAPS . 1 Introduction Figure 1: Illustration of MAPS , a framework for inferring the functionality of attention heads in LLMs from their parameters. MAPS casts the head as a matrix M which assigns a score for every pair of tokens in the model’s vocabulary. Then, it considers groups of token pairs (sub-matrices in M ) to measure how strongly the head implements a given operation (A) and to inspect the head’s salient operations (B) . Attention heads play a key role in modern large language models (LLMs) ( Vaswani et al., 2017 ; Zhou et al., 2024 ; Olsson et al., 2022 ) . Numerous studies ( Zheng et al., 2024 ; Ferrando et al., 2024 ) have explored their functionality, typically by analyzing their attention patterns or outputs during inference for certain inputs or tasks. However, relying on the model’s behavior for certain inputs has drawbacks. First, this approach may overlook some of the functions implemented by the head, as heads can exhibit different behaviors for different inputs ( Gould et al., 2024 ; Merullo et al., 2024a ; Olsson et al., 2022 ; Kissane et al., 2024 ) . Second, a comprehensive analysis of the head’s operation would require executing the model over numerous inputs, potentially the whole training corpus, which involves a high computational cost and could be impossible when the data is unavailable. Last, analyzing the examples that activate the head is often non-trivial and could be misleading ( Bolukbasi et al., 2021 ; Gao et al., 2024 ; Kissane et al., 2024 ) . In this work, we consider a different approach to this problem, where our goal is to infer the functionality of attention heads directly from their parameters and without executing the model. To this end, we leverage the approach of interpreting model parameters in the vocabulary space ( Geva et al., 2021 ; Geva et al., 2022 ; Katz et al., 2024 ) . Specifically, we build on the formulation by Elhage et al. (2021) ; Dar et al. (2023) , who cast the attention head as a matrix M , where each entry is a mapping score between two tokens. While this approach has been shown effective in identifying heads with certain operations, so far its usage has been limited to studying specific heads in detected circuits Wang et al. (2023) ; McDougall et al. (2024) or a single operation Gould et al. (2024) . Here, we scale this interpretation approach into a general framework, called MAPS (Mapping Attention heads ParameterS), which enables answering two types of basic questions: (a) given a predefined operation, mapping how strongly different heads across the model implement it, and (b) given an attention head, inferring its prominent operations. This is done by considering patterns across groups of mappings in M , as illustrated in Figure 1 . Predefined relations signify groups of mappings expressing a certain relation (e.g. city of a country or pronoun resolving). Salient operations consist of subsets of mappings for which the head induces the most prominent effect. In addition, analyzing simple statistics of these mappings provides insights into how global or specific its operation is. We evaluate our framework on 6 popular LLMs and 20 predefined relations of 4 categories – knowledge, language, algorithmic, and translation. Experiments show that estimations by MAPS strongly correlate with the head outputs during inference. Moreover, causally removing all the heads implementing a certain operation substantially impairs the model’s ability to answer queries requiring this operation, compared to removing other heads. Analysis of the obtained mappings shows that, across all models, MAPS detects relation heads mostly in the middle and upper layers, while revealing universality patterns for several relations. Moreover, it demonstrates how the model’s architecture introduces biases in function encoding. Smaller models tend to encode higher numbers of relations on a single head, and in Llama-3.1 models, which use grouped-query attention, grouped attention heads often implement the same or similar relations. Notably, MAPS successfully detected previously identified heads of specific operations, while discovering additional heads of similar operations not reported before. Next, we demonstrate the utility of MAPS for inferring the prominent operations of a given head. We consider the head’s salient mappings in M and use GPT-4o Hurst et al. (2024) to automatically describe the functionality they exhibit. Applying this procedure to GPT-2 xl and Pythia 6.9B, we map the prominent operations of 62% of their heads and 60%-96% of those in the middle and upper layers. Qualitative analysis shows semantic, linguistic, and algorithmic operations and reveals novel operations, such as the extension of time periods ( day->month;month->year ). A human study shows that our automated pipeline performs reasonably well, and GPT-4o reliably detects observable operations. To conclude, we introduce MAPS , an efficient framework for inferring attention heads’ functionality from their parameters. We showcase the utility of MAPS in systematically mapping a certain functionality across the model and automatically characterizing the salient operations of a given head. Estimations by MAPS correlate with the head’s outputs and are faithful to the model’s behavior, and provide valuable insights on architecture biases and universality of head operations in LLMs. 2 Preliminaries and Notation We assume a transformer-based LM with a hidden dimension d , L layers, H attention heads per layer, a vocabulary \mathcal{V} , an embedding matrix E\in\mathbb{R}^{|\mathcal{V}|\times d} , and an unembedding matrix U\in\mathbb{R}^{d\times|\mathcal{V}|} . Attention heads as interaction matrices We use the formulation by Elhage et al. (2021) and view an attention head as two “interaction” matrices W_{QK},W_{VO}\in\mathbb{R}^{d\times d} . Given a sequence of n hidden states X\in\mathbb{R}^{n\times d} , the matrix W_{QK} computes the query-key scores to produce an attention weights matrix A\in\mathbb{R}^{n\times n} : A=\text{softmax}\Bigg(\frac{X(W_{QK})X^{T}}{\sqrt{d/H}}\Bigg) The matrix W_{VO} operates on the contextualized hidden states according to A , namely \tilde{X}=AX , and produces the head’s output Y\in\mathbb{R}^{n\times d} : Y=\tilde{X}W_{VO} (1) The matrix W_{QK} can be viewed as “reading”

Read the complete text