Cross-Layer Discrete Concept Discovery for Interpreting Language Models

Ankur Garg, Xuemin Yu, Hassan Sajjad & Samira Ebrahimi Kahou · 2025

Read Cross-Layer Discrete Concept Discovery for Interpreting Language Models by Ankur Garg, Xuemin Yu, Hassan Sajjad & Samira Ebrahimi Kahou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20040.txt.

Introduction

Ankur Garg, Xuemin Yu, Hassan Sajjad, and Samira Ebrahimi Kahou seek the ideas inside a language model in a 2025 paper. The stream that runs through such a model mixes and copies traits from layer to layer, so a look at one layer misses much.

Tools that span layers work in a smooth space where one idea can smear across many cells. The new frame maps one layer to the next through a gate of set codes, which packs the smeared traits into crisp idea marks. A smart pick rule and a slow drift of the code book keep the marks rich and wide. On three text sets the frame beats past tools. Cut its ideas out and test scores crash. Judges rank its ideas first in most duels. People can read its maps to guess the model's calls, far more often than with past tools.

Catalog record

CREATORAnkur Garg, Xuemin Yu, Hassan Sajjad & Samira Ebrahimi Kahou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20040

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-Layer Discrete Concept Discovery for Interpreting Language Models Ankur Garg, Xuemin Yu, Hassan Sajjad & Samira Ebrahimi Kahou Abstract Uncovering emergent concepts across transformer layers remains a significant challenge because the residual stream linearly mixes and duplicates information, obscuring how features evolve within large language models. Current research efforts primarily inspect neural representations at single layers, thereby overlooking this cross-layer superposition and the redundancy it introduces. These representations are typically either analyzed directly for activation patterns or passed to probing classifiers that map them to a limited set of predefined concepts. To address these limitations, we propose cross-layer VQ-VAE (CLVQ-VAE) , a framework that uses vector quantization to map representations across layers and in the process collapse duplicated residual-stream features into compact, interpretable concept vectors. Our approach uniquely combines top- k k temperature-based sampling during quantization with EMA codebook updates, providing controlled exploration of the discrete latent space while maintaining code-book diversity. We further enhance the framework with scaled-spherical k-means++ for codebook initialization, which clusters by directional similarity rather than magnitude, better aligning with semantic structure in word embedding space. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks, yet their internal mechanisms remain largely opaque. This opacity poses major challenges for interpretability, limiting our scientific understanding and raising concerns around trust, accountability, and responsible use ( Dodge et al. 2021 ; Sheng et al. 2021 ) . Current interpretability work typically analyzes representations at individual layers, either through direct examination of activation patterns ( Zhang et al. 2021 ) or by employing probing classifiers that map hidden states to predefined concepts ( Belinkov et al. 2017 ; Arps et al. 2022 ; Kumar et al. 2023 ) . Such single-layer views ignore how transformer residual streams duplicate and mix information across layers, masking the true computational structure that emerges only when these layers are considered jointly ( Team 2024 ) . Recent advances in sparse autoencoder (SAE) methods ( Härle et al. 2024 ; Lan et al. 2025 ) and transcoder architectures ( Marks et al. 2024 ; Dunefsky et al. 2024a ) highlight the value of analyzing layer pairs. Empirical studies show that cross-layer analysis often yields more interpretable features than per-layer approaches ( Shi et al. 2025 ; Balagansky et al. 2025 ; Laptev et al. 2025 ) . This is largely due to the additive residual stream in transformers: each layer contributes to the running representation, causing features to persist and appear duplicated when layers are viewed in isolation ( Lindsey et al. 2025 ) . However, SAE -based methods operate in continuous spaces, requiring arbitrary thresholds to interpret sparse activations. This continuity limits interpretability, as the resulting neurons lack discrete conceptual boundaries aligned with human categorical reasoning ( Wu et al. 2024 ) . Vector quantized-variational autoencoder (VQ-VAE) have been extensively explored in computer vision to discretize continuous representations into codebook vectors ( van den Oord et al. 2018 ; Takida et al. 2022 ; Razavi et al. 2019 ) . Although VQ-VAEs share the reconstruction objective of SAEs , they critically differ by mapping inputs to discrete codebook vectors rather than continuous activations. When applied to NLP, these codebook vectors can be interpreted as linguistic concepts that are essential for reconstruction. Building on these insights, we propose CLVQ-VAE , a framework that discovers concepts between transformer layers. Unlike standard VQ-VAEs which reconstruct the same layer, our model acts as a transcoder, mapping activations from a lower layer l to a higher layer h through a discrete bottleneck, hence collapsing redundant residual-stream features into interpretable codebook vectors. We enhance this architecture with two key technical contributions: (1) a stochastic sampling mechanism that selects from top-k nearest codebook vectors using temperature-controlled probability distributions, significantly improving codebook utilization and concept diversity compared to deterministic approaches; and (2) directional similarity-based codebook initialization through scaled-spherical k-means++, which better captures the semantic structure of language embeddings where angular relationships often carry more meaning than Euclidean distances. We evaluate CLVQ-VAE on ERASER ( Pang & Lee 2004 ) , Jigsaw Toxicity ( cjadams et al. 2017 ) , and AGNEWS ( Gulli 2005 ) using fine-tuned RoBERTa ( Liu et al. 2019b ) , BERT ( Devlin et al. 2019 ) , and LLaMA-2-7b models. Perturbation-based experiments show that our approach identifies salient concepts that strongly influence predictions, outperforming clustering, single-layer VQVAE, and SAE baselines. Human evaluation corroborates these findings, with CLVQ-VAE visualisations achieving markedly higher model-alignment and inter-annotator agreement scores than clustering approach, indicating that our discrete concepts faithfully interprets the reason of the prediction. 2 Methodology We propose CLVQ-VAE , a modular framework for cross-layer concept discovery in transformers that aligns lower-layer quantized representations with higher-layer activations. As shown in Figure 1 , the framework comprises three main components: (i) an adaptive residual encoder that applies minimal, learnable transformations to lower layer embeddings via a controllable interpolation mechanism; (ii) a vector quantizer that discretizes encoder outputs into concept vectors; and (iii) a transformer decoder that reconstructs higher-layer representations from the quantized concepts. The model takes hidden states from a lower transformer layer (e.g., Layer 8), processes them through the encoder to preserve semantic content while enabling controlled transformation, and then maps the result to a discrete latent space via the quantizer. These concept vectors are decoded to approximate a higher-layer activation (e.g., Layer 12). This training is guided by a reconstruction loss that encourages the model to capture persistent, distributed features across layers. Figure 1: Overview of the CLVQ-VAE framework for cross-layer concept discovery. Lower-layer activations are passed through an adaptive residual encoder, discretized via vector quantization into concept vectors, and decoded to reconstruct higher-layer representations. 2.1 Adaptive Residual Encoder Transformer-based language models produce rich, contextualized embeddings at each layer that have been shown to encode diverse linguistic information ( Liu et al. 2019a ; Sajjad et al. 2022b ) . A major challenge in designing the encoder is to ensure the preservation of semantic knowledge present in the input embeddings while allowing for subtle transformations that facilitate effective cross-layer concept mapping. We propose an adaptive residual encoder that implements a controlled manipulation of input embedding. Rather than completely transforming the input, which destroy valuable linguistic features due to random initialization of encoder parameters, or leaving it unchanged, which would limit concept discovery, we introduce a learnable interpolation mechanism. This design respects the information-rich nature of pre-trained language model embeddings while enabling targeted refinements for cross-layer concept identification. Given an input embedding \mathbf{x}\in\mathbb{R}^{d} from layer l , the encoder produces an output \mathbf{z}_{e} through a weighted mixture: \mathbf{z}_{e}=(1-\alpha)\cdot\mathbf{x}+\alpha\cdot\text{LN}(W\mathbf{x}+\mathbf{b}) (1) where \alpha=\sigma(a)*0.5 is a learnable mixing coefficient constrained to [0,0.5] and LN is the layer normalization applied to the linearly transformed x . We limit \alpha to a maximum of 0.5 to prevent excessive modification of the original embedding, as we empirically found in Table 8 that allowing complete transformation ( \alpha\in [0,1]) resulted in reduced codebook utilization. This architecture ensures that the original input can be adaptively modified, preserving the foundation of the representation while allowing the model to learn which aspects to adjust. In Table 8 , we observe that for an adaptive alpha, the model initially prefers

Read the complete text