Layers at Similar Depths Generate Similar Activations Across LLM Architectures

Christopher Wolfram, Aaron Schein · 2025

Read Layers at Similar Depths Generate Similar Activations Across LLM Architectures by Christopher Wolfram, Aaron Schein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08775.txt.

Introduction

Christopher Wolfram and Aaron Schein compare the inner activation patterns of twenty-four open language models, in this 2025 arXiv paper. The question is whether models trained apart still organize meaning alike. The pair tracked, layer by layer, which inputs land near which inside each model's hidden space.

Neighbor relations shifted as depth grew within every model, so no single map ruled all layers. Yet matching layers of different models shared roughly the same neighbor relations. The picture that emerges is a common journey: each model walks a similar sequence of inner geometries, stretched or squeezed to fit its own shape. The authors say the shared path hints at something universal in how these systems learn.

Catalog record

CREATORChristopher Wolfram, Aaron Schein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Layers at Similar Depths Generate Similar Activations Across LLM Architectures Christopher Wolfram, Aaron Schein Abstract How do the latent spaces used by independently-trained LLMs relate to one another? We study the nearest neighbor relationships induced by activations at different layers of 24 open-weight LLMs, and find that they 1) tend to vary from layer to layer within a model, and 2) are approximately shared between corresponding layers of different models. Claim 2 shows that these nearest neighbor relationships are not arbitrary, as they are shared across models, but Claim 1 shows that they are not “obvious” either, as there is no single set of nearest neighbor relationships that is universally shared. Together, these suggest that LLMs generate a progression of activation geometries from layer to layer, but that this entire progression is largely shared between models, stretched and squeezed to fit into different architectures. 1 Introduction Instead of trying to understand the internal mechanisms of specific language models, we study structural similarities in the activations generated by models of different architectures. In particular, how do the latent spaces used by different models relate to one another, and do they have any universal properties? Consider the following experiment. Let \mathcal{D} be a set of prompts. We pass each prompt in \mathcal{D} to a language model M and collect the activations generated at a particular layer 1 1 1 We use the term “layer” to refer to an entire decoder module. See Section 3.2 for details on how we extract activations from transformer models. \ell . For any prompt t\in\mathcal{D} , we can then ask: which are the k=10 other prompts that yielded activations most similar to this one? This gives a set of the k nearest neighbors of t among the other elements of \mathcal{D} . Repeating this for all prompts in \mathcal{D} gives a collection of nearest neighbor relationships that encode which prompts are nearest to which others according to layer \ell in the model M . By comparing nearest neighbor relationships across a wide range of models, we make two findings that seem unremarkable on their own, but together are quite surprising: Claim 1. Activations collected at different depths within the same model tend to have different nearest neighbor relationships. Claim 2. Activations collected at corresponding depths of different models tend to have similar nearest neighbor relationships. Item 2 shows that nearest neighbor relationships are not arbitrary, as they are shared across independently-trained models. Item 1 , however, shows that they are not “obvious” either, as nearest neighbor relationships vary with depth. Put another way, Item 1 shows that nearest neighbor relationships are sensitive to differences between layers within a single model, while Item 2 shows that activations at corresponding depths of different models are similar. Such nearest neighbor relationships reflect the local geometry of the embeddings generated at a particular layer, and are closely related to operations used in transformer models. Together, Items 1 and 2 suggest that transformer-based language models use a progression of distinct local geometries from layer to layer, and that this entire progression of local geometries is (largely) shared across models regardless of architecture. In other words, independently-trained language models tend to go through a similar sequence of intermediate representations when given the same prompt. Llama-3.1-8B Layer 10 Layer 30 Gemma-2-9B Layer 20 Layer 40 similar similar different different Figure 1 : Layers at different depths generate activations with different nearest neighbor relationships ( Item 1 ), but layers at corresponding depths of different models generate activations with similar nearest neighbor relationships ( Item 2 ). The particular example used in Sections 1 and 2 is shown here. To give a concrete example, let t be this text from OpenWebText ( Gokaslan et al. 2019 ) , a dataset mostly containing internet news articles: Training For Ice and Mixed Climbing Series brought to you by Furnace Industries continues... According to layer 10 of Llama-3.1-8B , these are the 3 nearest neighbors to that text t among a sample of 2047 other texts from OpenWebText: • There have been a lot of crazy reactions from the democrats now that Donald Trum... • Prosecutor Christina Weilander filed the charges with Södertörn district court o... • 7 questions on the Great British Bake Off Great British Bake Off Quiz Then there... At first glance, these nearest neighbors look arbitrary, and it is not clear what relationship they have with the original input or with one another. That makes it all the more surprising that layer 20 of Gemma-2-9B gives exactly the same 3 nearest neighbors (in line with Item 2 ). Moreover, layer 30 of Llama-3.1-8B gives another completely different set of nearest neighbors (in line with Item 1 ), but many of them are the same as the nearest neighbors given by layer 40 of Gemma-2-9B ( Fig. 1 ). Llama-3.1-8B and Gemma-2-9B are architecturally different models independently trained by different companies, so it is surprising that we not only find similarities in their latent representations, but that these similarities only appear at corresponding depths. (See Section 2 for more investigation of this specific example.) We study this phenomenon systematically by generating the n_{1}\times n_{2} affinity matrix , which stores the similarity of nearest neighbor relationships implied by each pair of layers (i,j) with layer i\in[n_{1}] from model M_{1} and layer j\in[n_{2}] from model M_{2} . The resulting affinity matrices exhibit diagonal structure, with higher similarity among layers at corresponding depths (implying Item 2 ), and lower similarity among pairs of layers from different depths (implying Item 1 ). This paper studies the emergence of diagonal structure in affinity matrices across 24 open-weight language models, with sizes spanning from 1B to 70B parameters. We find that diagonal structure is present when comparing a wide range of models ( Section 4 ), and that this diagonal structure is statistically significant for every pair of models we consider ( Section 4.1 ). 1.1 Related work There is an extensive literature on representational similarity in neural networks ( Kornblith et al. 2019 ; Klabunde et al. 2024 ) , and with rising interest in universality in LLMs ( Olah et al. 2020 ; Rai et al. 2025 ) more recent work has begun to apply representational similarity methods to large transformer models ( Huh et al. 2024 ; Klabunde et al. 2023 ) . However, none have systematically studied affinity matrices comparing layers of architecturally different LLMs or the resulting diagonal structure. Most either 1) compared layers within a single model ( Vulić et al. 2020 ) , 2) compared layers between two copies of a model trained with different random seeds ( Nguyen et al. 2022 ; Gurnee et al. 2024 ) , 3) focused on CNNs or very small language models ( Raghu et al. 2017 ) , 4) used a representational similarity measure that did not reveal diagonal structure ( Lan et al. 2024 ; Baek et al. 2024 ; Brown et al. 2023 ) , or some combination thereof ( Kornblith et al. 2019 ; Nguyen et al. 2021 ) . Of particular note is Wang et al. 2025 , who used sparse autoencoders to study representational similarity of Pythia-160M and Mamba-130M and noted that

Read the complete text