LLM Interpretability with Identifiable Temporal-Instantaneous Representation

Xiangchen Song, Jiaqi Sun, Zijian Li, Yujia Zheng, Kun Zhang · 2025

Read LLM Interpretability with Identifiable Temporal-Instantaneous Representation by Xiangchen Song, Jiaqi Sun, Zijian Li, Yujia Zheng, Kun Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23323.txt.

Introduction

Sparse autoencoders read language model features, but they ignore how concepts relate over time. Xiangchen Song and 4 co-authors add causal structure with theoretical guarantees. Their framework learns temporal causal representations tuned to the model's huge concept space. Both delayed and instant causal links are captured. Prior causal methods could not scale to that space.

Synthetic tests at real-world scale validate the approach first. The team then extend autoencoder techniques with the framework on real activations. Meaningful concept relationships surface inside the model. Guarantees come from identifiability results in causal representation learning. The authors argue that time-aware structure deepens interpretability beyond static features.

Catalog record

CREATORXiangchen Song, Jiaqi Sun, Zijian Li, Yujia Zheng, Kun Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-23323

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM Interpretability with Identifiable Temporal-Instantaneous Representation Xiangchen Song, Jiaqi Sun, Zijian Li, Yujia Zheng, Kun Zhang Abstract Despite Large Language Models’ remarkable capabilities, understanding their internal representations remains challenging. Mechanistic interpretability tools such as sparse autoencoders (SAEs) were developed to extract interpretable features from LLMs but lack temporal dependency modeling , instantaneous relation representation , and more importantly theoretical guarantees —undermining both the theoretical foundations and the practical confidence necessary for subsequent analyses. While causal representation learning (CRL) offers theoretically-grounded approaches for uncovering latent concepts, existing methods cannot scale to LLMs’ rich conceptual space due to inefficient computation. To bridge the gap, we introduce an identifiable temporal causal representation learning framework specifically designed for LLMs’ high-dimensional concept space, capturing both time-delayed and instantaneous causal relations. Our approach provides theoretical guarantees and demonstrates efficacy on synthetic datasets scaled to match real-world complexity. By extending SAE techniques with our temporal causal framework, we successfully discover meaningful concept relationships in LLM activations. Our findings show that modeling both temporal and instantaneous conceptual relationships advances the interpretability of LLMs. 2 2 footnotetext: Equal contribution. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language tasks, from question answering to content generation. Despite these achievements, a fundamental understanding of their internal representations remains underexplored. This gap between performance and interpretability poses significant challenges for ensuring the reliability, safety, and appropriate deployment of these increasingly powerful systems ( scholkopf2021toward ) . Mechanistic interpretability (MI) aims to bridge this gap by reverse-engineering neural networks to understand how they process and represent information ( cunningham2023sparseautoencodershighlyinterpretable ) . Among all MI tools, sparse autoencoders (SAEs) have emerged as a promising approach for extracting interpretable features from LLMs ( cunningham2023sparseautoencodershighlyinterpretable ; anthropic2024scaling ) . By decomposing the high-dimensional activations of LLMs into sparse, monosemantic features, SAEs help identify the basic units of computation within these complex systems. However, SAEs present several limitations that restrict their utility for comprehensive model understanding: First, SAEs treat each feature as an isolated representation, failing to capture how features influence one another. This omission disregards semantic connections and transitions within a sequence, which are known as temporal or time-delayed relationships between features * * * Alternative approaches, such as ameisen2025circuit ; lindsey2025biology , use attention scores from the LLM to infer time-delayed influence. . Second, SAEs lack mechanisms to represent instantaneous or logical relationships between features, such as mutual exclusivity or co-occurrence constraints ( lippe2022citris ) . These relationships complement the temporal dynamics by encoding structural dependencies within the same time step. Third, and most critically, SAEs offer no theoretical guarantees of the uniqueness of the recovered features. This absence undermines confidence that the extracted features reflect meaningful and stable latent variables, rather than arbitrary or unstable transformations von2024identifiable . Fortunately, to address these limitations, the causal representation learning (CRL) community has proposed a range of promising frameworks with theoretical guarantees scholkopf2021toward . For instance, ( li2025on ) and ( lippe2023causal ) use sparse causal influence and interventions to uncover temporal and instantaneous relationships among latent variables. However, these methods face significant scalability challenges due to the computational inefficiency of estimating Jacobians. As a result, they typically scale to only dozens or hundreds of concepts yao2022temporally , while interpretability in LLMs demands efficient modeling of thousands or even tens of thousands of concept features anthropic2024scaling . In summary, although CRL offers strong theoretical guarantees for recovering meaningful features and their causal relationships, its limited scalability in high-dimensional settings remains a major obstacle to practical deployment in LLM analysis. To bridge this gap, in this paper, we introduce a computationally efficient temporal causal representation learning framework specifically designed for the high-dimensional activation space in LLMs. Our approach builds upon recent advances in both sparse autoencoders for LLMs and causal representation learning for sequential data. The key contributions of our work are: (1) We propose a simple yet effective framework that jointly models time-delayed causal relations between concepts and instantaneous constraints, providing a more comprehensive understanding of how information flows through LLMs. (2) Leveraging sparsity principle, we establish theoretical guarantees for our approach, making the representations learned reliable and explainable. (3) Grounded in the theoretical result, we design scalable and efficient algorithms tailored to the high-dimensional concept space of LLMs, significantly extending prior work in CRL. (4) We validate our approach on synthetic datasets scaled to match real-world complexity and demonstrate its effectiveness when applied to activations from real LLMs. 2 Problem Setting Figure 1 : Graphical illustration of the data generation process. We begin by characterizing the generation process of LLM activations to establish interpretability guarantees. These activations—signals produced during inference—are widely assumed to be linearly generated from hidden concepts, consistent with sparse autoencoder (SAE) literature ( anthropic_sae_2024 ; gao2024scaling ) . However, existing formulations typically treat these concepts as independent, overlooking dependencies between them. In reality, earlier-token semantics often influence later tokens, and token generation depends jointly on the activation of multiple concepts. To account for these interactions, we introduce a data generation process with both temporal and instantaneous relations, adopting CRL terminology. Given a token sequence \boldsymbol{s}=(v_{1},\dots,v_{k}) , let \mathbf{x}_{t}=(x_{t,1},\dots,x_{t,m}) be the n -dimensional activation vector at token v_{t} for a specific layer. Following the linear representation hypothesis ( park2023linear ) and SAE formulation ( anthropic_sae_2024 ; gao2024scaling ) , we assume: \mathbf{x}_{t}=\mathbf{g}(\mathbf{z}_{t}), (1) where \mathbf{g}:\mathbb{R}^{n}\to\mathbb{R}^{m} is the linear mixing function, and \mathbf{x}_{t} and \mathbf{z}_{t} are observed and latent variables, respectively. Besides, each latent variable z_{t,i} is governed by a structural equation model (SEM) capturing both time-delayed and instantaneous dependencies: \begin{split}z_{t,i}=\sum_{\tau}\sum_{j\in\mathcal{J}_{i,\tau}}B_{i,j,\tau}z_{t-\tau,j}+\sum_{j\in\mathcal{K}_{i}}M_{i,j}z_{t,j}+\epsilon_{t,i},\end{split} (2) where B_{i,j,\tau} represents the coefficient for the time-delayed effect from z_{t-\tau,j} to z_{t,i} ; \mathcal{J}_{i,\tau} is the set of indices of latent variables that have a time-delayed effect on z_{t,i} with lag \tau ; M_{i,j} represents the coefficient for the instantaneous effect from z_{t,j} to z_{t,i} ; \mathcal{K}_{i} is the set of indices of latent variables that have an instantaneous effect on z_{t,i} ; and \epsilon_{t,i} denotes the temporally and spatially independent noise extracted from a distribution p_{\epsilon_{i}} . The graphical model for this process is illustrated in Figure 1 . To better understand this data generation process in the context of LLM activations, \mathbf{x}_{t} represents activations in a specific layer l for token v_{t} , and the latent variables \mathbf{z}_{t} can be considered as the underlying causal factors that generate these activations. In this case, the instantaneous effects (coefficients M_{i,j} ) reflect semantic or syntactic relationships between different latent factors within the same token, while time-delayed effects (coefficients B_{i,j,\tau} ) represent dependencies on previous tokens. Putting them together, the underlying data generating process can be written as \underbrace{\mathbf{x}_{t}=\mathbf{g}(\mathbf{z}_{t})}_{\text{Linear mixing}},\quad\underbrace{z_{t,i}=\sum_{\tau>0}\sum_{j\in\mathcal{J}_{i,\tau}}B_{i,j,\tau}z_{t-\tau,j}+\sum_{j\in\mathcal{K}_{i}}M_{i,j}z_{t,j}+\epsilon_{t,i},\;i=1,\dots,m}_{\text{Linear latent temporal SEM}}. (3) The linear latent temporal SEM in Eq. ( 3 ) induces two types of causal relationships: • A time-delayed causal graph \mathcal{G}_{d} with vertices \{z_{t,i}\}_{i=1}^{n} across different token positions and edges z_{t-\tau,j}\rightarrow z_{t,i} if and only if B_{i,j,\tau}\neq 0 . • An instantaneous causal graph \mathcal{G}_{e} with vertices \{z_{t,i}\}_{i=1}^{n} at each token position t and edges z_{t,j}\rightarrow z_{t,i} if and only if M_{i,j}\neq 0 . We assume that \mathcal{G}_{e} is acyclic, i.e., a directed acyclic graph (DAG). This implies that the matrix M can be permuted to be strictly lower triangular,

Read the complete text