How LLMs Learn: Tracing Internal Representations with Sparse Autoencoders

Tatsuro Inaba, Kentaro Inui, Yusuke Miyao, Yohei Oseki, Benjamin Heinzerling, Yu Takagi · 2025

Read How LLMs Learn: Tracing Internal Representations with Sparse Autoencoders by Tatsuro Inaba, Kentaro Inui, Yusuke Miyao, Yohei Oseki, Benjamin Heinzerling, Yu Takagi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06394.txt.

Introduction

Tatsuro Inaba and seven co-authors wrote this 2025 arXiv paper on how a bilingual language model organizes its two languages inside. The team opened up the model with sparse autoencoders, tools that split tangled internal activity into cleaner, named features. They tracked those features across training time, across the model's layers, and across model sizes.

Early in training, the model kept its languages apart, handling each on its own. Alignment between them grew later, strongest in the middle layers where meaning is usually thought to live. Bigger models aligned more strongly. To prove the alignment matters, the team grafted features from a finished model into one still training and watched performance respond. The study offers a rare inside view of how multilingual skill assembles itself, with lessons for training models that must serve many languages well.

Catalog record

CREATORTatsuro Inaba, Kentaro Inui, Yusuke Miyao, Yohei Oseki, Benjamin Heinzerling, Yu Takagi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How LLMs Learn: Tracing Internal Representations with Sparse Autoencoders Tatsuro Inaba, Kentaro Inui, Yusuke Miyao, Yohei Oseki, Benjamin Heinzerling, Yu Takagi Abstract Large Language Models (LLMs) demonstrate remarkable multilingual capabilities and broad knowledge. However, the internal mechanisms underlying the development of these capabilities remain poorly understood. To investigate this, we analyze how the information encoded in LLMs’ internal representations evolves during the training process. Specifically, we train sparse autoencoders at multiple checkpoints of the model and systematically compare the interpretative results across these stages. Our findings suggest that LLMs initially acquire language-specific knowledge independently, followed by cross-linguistic correspondences. Moreover, we observe that after mastering token-level knowledge, the model transitions to learning higher-level, abstract concepts, indicating the development of more conceptual understanding. 1 1 1 Our code, demo, and SAE weights are publicly available at https://github.com/llm-jp/llm-jp-sae . 1 Introduction Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks, from multilingual translation to advanced semantic understanding Bubeck et al. 2023 . As these models become increasingly complex and widespread, the need to understand their internal mechanisms has grown significantly. This has fueled a surge of research aimed at interpreting their mechanisms and decision-making processes, leading to intriguing insights into their behavior Casper et al. 2023 ; Bereska and Gavves 2024 . However, fundamental questions regarding how LLMs acquire and develop these capabilities remain poorly understood. For instance, do LLMs learn language-specific concepts independently, or do they simultaneously acquire cross-lingual concepts that generalize across languages? Similarly, is there a prioritization in learning low-level, token-specific features versus high-level, abstract concepts? Figure 1: Illustration of our approach to comparing internal representations across different training stages of LLMs. We train SAEs on the internal representation from multiple checkpoints. In this work, we address this gap by analyzing how the information encoded in the internal representations of LLMs evolves over time. Specifically, we employ sparse autoencoders (SAEs) Bricken et al. 2023 ; Huben et al. 2024 to analyze the hidden representations from multiple checkpoints of a large language model. By examining the distribution of SAE features at each checkpoint, we identify the types of information the model encodes at different training stages of its development (see Figure 1 ). Our experiments yield two key findings: (1) LLMs first learn knowledge within individual languages before acquiring cross-lingual mappings (§ 4.3 ), and (2) they initially capture fine-grained, token-level knowledge before progressing to more abstract, conceptual representations (§ 4.4 ) These findings offer new insights into the internal mechanisms that underlie the emergence of LLMs’ generalization abilities. 2 Sparse Autoencoders A sparse autoencoder (SAE) is an autoencoder that enforces a sparsity constraint on its hidden layer. In this study, we adopt a variant called TopK-SAE Makhzani and Frey 2014 , where the TopK activation function is applied at the hidden layer. Compared to a ReLU-based SAE Bricken et al. 2023 ; Huben et al. 2024 , TopK-SAE has been shown to be easier to train while maintaining sparsity and achieving higher reconstruction performance Gao et al. 2025 . Let x\in\mathbb{R}^{d} be the input vector and n be the dimension of the hidden layer. The encoder and decoder are defined as follows: \displaystyle z \displaystyle=\mathrm{TopK}\bigl(W_{\mathrm{enc}}(x-b_{\mathrm{pre}})\bigr), (1) \displaystyle\hat{x} \displaystyle=W_{\mathrm{dec}}z+b_{\mathrm{pre}}, (2) where W_{\mathrm{enc}}\in\mathbb{R}^{n\times d} and W_{\mathrm{dec}}\in\mathbb{R}^{d\times n} are learned linear layers, and b_{\mathrm{pre}}\in\mathbb{R}^{d} is a learnable bias parameter. W_{\mathrm{dec}} is initialized as the transpose of W_{\mathrm{enc}} , and b_{\mathrm{pre}} is initialized to the geometric median of the input data. The training objective is the following mean squared error (MSE) loss: \displaystyle L=\|x-\hat{x}\|^{2}_{2}. (3) Two hyperparameters control TopK-SAE. In this study, we control TopK-SAE by two hyperparameters: n , the dimension of the hidden layer, and K , the number of hidden dimensions to keep active. Interpreting W_{\mathrm{dec}} as n distinct vectors in \mathbb{R}^{d} , TopK-SAE can be seen as selecting K vectors from n and using their weighted sum to reconstruct the input. In this study, we denote each dimension of the encoder output z\in\mathbb{R}^{n} as a feature . When a feature is selected in the top-K operation and used in reconstruction, we say the feature is activated . 3 Preliminary Experiments Figure 2: Reconstruction loss for varing hidden dimensions n and the sparsity K . Larger n and K improve reconstruction accuracy. We begin by conducting preliminary experiments on a pre-trained LLM to tune SAE hyperparameters and validate the interpretability of resulting features. 3.1 Experimental Setup We use the 12th layer output ( d=2048 ) of the 24-layer model llm-jp-3-1.8B 2 2 2 https://huggingface.co/llm-jp/llm-jp-3-1.8b Aizawa et al. 2024 as the input to the TopK-SAE. The model is trained on the LLM-jp Corpus v3 3 3 3 https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-corpus-v3 , which contains a total of 1.7T tokens: 950B for English, 592B for Japanese, 114B for code, 0.8B for Korean, and 0.3B for Chinese. We selected the llm-jp-3-1.8B model because its intermediate checkpoints are (or will be) publicly accessible, it has over 1 billion parameters to exhibit emergent behaviors, and its training on both English and Japanese enables cross-lingual analysis. We train TopK-SAE with the Japanese and English Wikipedia subsets in the LLM-jp Corpus v3. For each document, we extract the first 65 tokens as the input to the LLM, discard the [BOS] token representation, and apply L2 normalization to the remaining 64 token embeddings, which serve as inputs to the SAE. The dataset consists of 165M tokens (50% Japanese, 50% English), split into 80% for training, 10% for validation, and 10% for testing. We fix the batch size at 32,768, use a warm-up phase of 1,000 steps, and perform a grid search to optimize the learning rate. Training a single SAE took about 1 hour using two A100 40GB GPUs. 3.2 Effect of Hyperparameters Figure 2 shows how the impact of varying the hidden dimensions n and the number of active dimensions K . Increasing either n or K reduces the reconstruction error. However, these hyperparameters significantly influence interpretability: if n or K is too large, a single concept may be fragmented into multiple features; if it is too small, multiple distinct concepts may be merged into a single feature. Identifying the optimal balance between reconstruction performance and interpretability remains an active area of research Menon et al. 2024 ; Leask et al. 2025 . Figure 3: Examples of feature activations across different training checkpoints. (a) Checkpoint 100, (b) Checkpoint 10,000, (c) Final checkpoint (988,240). Early in training, features activate on seemingly random fragments. As training progresses, features begin to capture language-specific or token-level meanings. By the final checkpoint, they encode higher-level cross-lingual semantics and abstract conceptual knowledge. Additional examples are provided in Figure 6 and in our demo 1 . 3.3 Patterns in Feature Activation Figure 3 (c) shows examples of feature activations ( n=32768 and K=32 ). The background color density indicates the magnitude of activation. For instance, \mathrm{F}_{\mathrm{ckpt}=988240}\#00009 strongly activates in segments defining certain terms, while \mathrm{F}_{\mathrm{ckpt}=988240}\#00016 activates in text about the smell, color, or state of a substance. These examples, along with others in Figure 6 (c), demonstrate that the features of TopK-SAE successfully capture semantically coherent and interpretable meanings. 4 Experiments In this section, we conduct main experiments, where we train SAEs on the internal representations of a large language

Read the complete text