Vector Arithmetic in Concept and Token Subspaces

Sheridan Feucht, Byron Wallace, David Bau · 2025

Read Vector Arithmetic in Concept and Token Subspaces by Sheridan Feucht, Byron Wallace, David Bau (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18162.txt.

Introduction

Sheridan Feucht, Byron Wallace, and David Bau show that language models keep meaning and surface form in separable parts of their internal activations. Earlier work found attention heads that copy word meanings and others that copy the literal token, and the authors use those heads to transform hidden states into concept and token subspaces.

In the concept subspace, classic analogy arithmetic works far better, with much higher nearest neighbor accuracy than raw activations. In the token subspace, similar arithmetic reveals surface relations, such as verb form changes. The results give a concrete way to read and manipulate what kind of information a model carries at a given point.

Catalog record

CREATORSheridan Feucht, Byron Wallace, David Bau
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18162

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vector Arithmetic in Concept and Token Subspaces Sheridan Feucht, Byron Wallace, David Bau Abstract In order to predict the next token, LLMs must represent semantic and surface-level information about the current word. Previous work identified two types of attention heads that disentangle this information: (i) Concept induction heads, which copy word meanings, and (ii) Token induction heads, which copy literal token representations ( 2 ) . We show that these heads can be used to identify subspaces of model activations that exhibit coherent semantic structure in Llama-2-7b. Specifically, when we transform hidden states using the attention weights of concept heads, we are able to more accurately perform parallelogram arithmetic ( 4 ; 5 ) on the resulting hidden states, e.g., showing that Athens – Greece + China = Beijing . This transformation allows for much higher nearest-neighbor accuracy (80%) than direct use of raw hidden states (47%). Analogously, we show that token heads allow for transformations that reveal surface-level word information in hidden states, allowing for operations like coding – code + dance = dancing . \workshoptitle Mechanistic Interpretability 1 Introduction Consider how an LLM might model the word boat . A boat is a type of vehicle that floats on water, can be powered by sails or engines, and generally carries one or more people. But there are many other important facts about this word: It is an English word that is all lowercase; it starts with the letter ‘b’; it rhymes with (and looks like) coat , and it is a singular common noun referring to a broad category. If we wish to analyze the relationship between the word boat and the word water , we must focus on the semantics of these words, discarding all of the other information that an LLM might encode. On the other hand, if we are trying to find a word that rhymes with boat , its meaning may not be particularly helpful to know. In the original word2vec paper, Mikolov et al. 4 ; 5 embed words in a manner that allows for parallelogram-like vector arithmetic: they claim that their embedding space is structured such that man is to woman as king is to queen . However, we find that their approach is only somewhat effective for raw Llama-2-7b hidden states ( 7 ) (Section 3 ). We hypothesize that these apparently poor results observed using a naive approach may be attributed to “interference” from irrelevant information in model activations. In other words, we posit that much of the information packed into LLM hidden states has nothing to do with semantics, and that word2vec arithmetic is only effective if performed in a semantic subspace of model activations. By using the weights of concept induction heads from 2 , we isolate a lower-dimensional space of Llama-2-7b activations for which, e.g., the representation of king - man + woman \approx queen . We also find that we can use token induction heads to perform parallelogram arithmetic for surface-level tasks, like identifying the first letter in a word, with much higher accuracy than using raw hidden states. This suggests that concept and token induction heads from 2 exhibit rich structure in their outputs, operating in subspaces of model activations that represent different facets of words. 2 Method: Concept and Token Lens In recent work, Feucht et al. 2 identify two types of attention heads responsible for copying text in-context: token induction heads (first described in 1 ), which copy exact tokens, and concept induction heads, which copy “fuzzy” word meanings. As these attention heads are responsible for copying previous words seen in-context, their value and output weights can be naturally viewed as transformations that extract semantic and token-level information from any given hidden state. Feucht et al. 2 use this insight to develop a “concept lens” that visualizes semantic information in hidden states. We repurpose their approach to derive general concept and token transformations that reveal meaningful structure in hidden states, in the sense that arithmetic in the resultant space accords with intuitive analogies. Let d be Llama-2-7b’s hidden dimension and m the dimension of a single head. We rely on a key insight from Elhage et al. 1 : the value and output projections for a particular head h at layer l , \smash{V_{(l,h)}\in\mathbb{R}^{(m,d)}} and \smash{O_{(l,h)}\in\mathbb{R}^{(d,m)}} respectively, are solely responsible for whatever information a head writes into the residual stream. Specifically, they point out that the product of these two matrices O_{(l,h)}V_{(l,h)} is a low-rank d\times d matrix (at most rank m ) that determines the effect of head (l,h) on the residual stream. In other words, multiplying a hidden state x_{l} by this matrix extracts whatever information within x_{l} that this head typically contributes to the residual stream. As described in 2 , to build a concept lens \smash{L_{C_{k}}\in\mathbb{R}^{(d,d)}} that reads from all of the concept induction head subspaces simultaneously, we combine the weights from the top- k concept induction heads C_{k} . We calculate the sum of the top- k concept OV matrices: L_{C_{k}}=\sum_{(l,h)\in C_{k}}O_{(l,h)}V_{(l,h)}. (1) If all attention heads in C_{k} are in the same layer, L_{C_{k}}x_{l} is mathematically equivalent to taking the sum of the outputs of those attention heads. However, we also allow for summation of heads across layers, which was empirically effective in prior work ( 6 ) , possibly because transformer representations are interchangeable in intermediate layers ( 3 ) . We can repeat this process using the top- k token induction heads T_{k} to obtain a token lens , which reveals information about the written form of a word. 3 Parallelogram Arithmetic 3.1 Approach We test the assertion made by Mikolov et al. 4 that embeddings should exhibit parallelogram-like structure: in other words, we test whether man – woman = king – queen . Figures 1 a and 1 b illustrate our approach. We use data from Mikolov et al. ( 4 ) and Todd et al. 6 , which consists of tuples of words in some relation to each other. For every possible pair of tuples, we want to evaluate whether the difference between one tuple is equal to the difference between another; i.e., for ( Athens , Greece ) and ( Beijing , China ), we want to evaluate whether Athens – Greece = Beijing – China . In general, we notate this as a-b=a^{\prime}-b^{\prime} for a pair of tuples ( a , b ) and ( a^{\prime} , b^{\prime} ). To obtain embeddings for each word w , we first pass that word through the model in a clean run, obtaining a single vector w_{\ell} by taking its last token representation at a particular layer \ell . We then transform this hidden state using some d\times d matrix L to obtain Lw_{\ell} . In the raw setting, we do not transform w_{\ell} at all, so L=I_{d} . In the concept setting, we use L=L_{C_{k}} , in the token setting, we use L=L_{T_{k}} , and as a baseline, we use L=L_{all} , which is the sum of all attention head OV matrices. Finally, to see whether a_{\ell}-b_{\ell}=a^{\prime}_{\ell}-b^{\prime}_{\ell} in the subspace mapped to by L , we calculate La_{\ell}-Lb_{\ell}+Lb^{\prime}_{\ell} and evaluate if La^{\prime}_{\ell} is its nearest neighbor among all possible words in this task.

Read the complete text