Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders

Shun Wang, Tyler Loakman, Youbo Lei, Yi Liu, Bohao Yang, Yuting Zhao, Dong Yang, Chenghua Lin · 2025

Read Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders by Shun Wang, Tyler Loakman, Youbo Lei, Yi Liu, Bohao Yang, Yuting Zhao, Dong Yang, Chenghua Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06427.txt.

Introduction

Shun Wang and seven co-authors use sparse code tools that learn on their own to open language models and lift their task scores. The word learn way pulls traits with one sense from nerve cells that most times mix more than one sense. With those traits in view, the team could spot wrong reads in the model of set tasks.

In a stand out point, those wrong reads could be set right by prompts writ once more on their own, with more notes. That led the model toward the read staff meant. The way by much lifted scores on tasks in the next steps. They were math thought and spot of when words stand for more than their plain sense, in tales and speech. The work hints that tools to read models are not just checks. They can feed back into more good inputs and more good scores.

Catalog record

CREATORShun Wang, Tyler Loakman, Youbo Lei, Yi Liu, Bohao Yang, Yuting Zhao, Dong Yang, Chenghua Lin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06427

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring Task Performance with Interpretable Models via Sparse Auto-Encoders Shun Wang, Tyler Loakman, Youbo Lei, Yi Liu, Bohao Yang, Yuting Zhao, Dong Yang, Chenghua Lin Abstract Large Language Models (LLMs) are traditionally viewed as black-box algorithms, therefore reducing trustworthiness and obscuring potential approaches to increasing performance on downstream tasks. In this work, we apply an effective LLM decomposition method using a dictionary-learning approach with sparse autoencoders. This helps extract monosemantic features from polysemantic LLM neurons. Remarkably, our work identifies model-internal misunderstanding, allowing the automatic reformulation of the prompts with additional annotations to improve the interpretation by LLMs. Moreover, this approach demonstrates a significant performance improvement in downstream tasks, such as mathematical reasoning and metaphor detection. 1 Introduction The interpretability of Large Language Models (LLMs) is a critical area of research. Despite their remarkable performance across a range of tasks, these models remain inherently opaque Li et al. 2024 . The ability to understand the internal decision-making processes and representational structures of LLMs is essential for building trust, ensuring safety, and optimizing performance, particularly in sensitive fields such as healthcare Al-Nazi and Peng 2024 and law Shu et al. 2024 . This requirement for interpretability stems from the black-box nature of neural networks, where the sheer complexity poses a significant challenge in understanding how input data is processed and represented within the model. Addressing this opacity is a key focus for advancing the responsible deployment of LLMs in real-world applications Lin et al. 2024 . LLMs such as GPT-4 Achiam et al. 2023 and Llama 3 Dubey et al. 2024 are trained on massive datasets to capture intricate language patterns. However, their opacity (i.e., their inability to be easily understood by humans) remains a significant limitation. Rudin 2019 argues against using black-box models in sensitive fields, advocating for inherently interpretable models that do not require post-hoc explanations which may not meet legal criteria for transparency Bordt et al. 2022 . Similarly, Lipton 2018 critiques the over-reliance on post-hoc interpretability techniques. These concerns have driven research on new methods for breaking down the internal mechanics of LLMs to understand how they process information. OpenAI proposed a method for using large models to interpret other large models Bills et al. 2023 . They utilized GPT-4 to analyze the internal structure of GPT-2 to reveal more information. However, as noted in their report, this simplistic approach primarily relies on analyzing how individual neurons respond to specific features in the data, which presents certain limitations. As discussed in Elhage et al. 2022 , neurons are often polysemantic, meaning they respond to multiple unrelated inputs, making it difficult to interpret their behavior based on individual activations alone Bricken et al. 2023 . Since each neuron is typically activated by multiple features, it is challenging to determine the precise proportion of activations attributed to each feature and their impact on the output. Furthermore, this limitation hinders the application of interpretability methods to improve downstream tasks, as the effect of activating a particular node is multifaceted, forming a one-to-many mapping. Consequently, while it is possible to infer the behavior of neurons from the output retrospectively, it remains difficult to predict the output state directly from the activation of individual neurons. To address this issue, inspired by the work of Bricken et al. 2023 , we apply a mechanistic interpretability technique that combines dictionary learning with sparse autoencoders to decompose polysemantic neurons in various LLMs. One possible explanation for the polysemantic nature of neurons is superposition Arora et al. 2018 ; Olah et al. 2020 ; Elhage et al. 2022 ; Xiong et al. 2024 , where a neural network represents more independent features than it has neurons by assigning each feature a unique linear combination of neurons. In this scenario, features can be considered vectors spanning multiple neurons, forming an overcomplete basis for activations. Prior studies on toy models Elhage et al. 2022 demonstrated that when features in the data are sparse, the training process may encourage the model to utilize superposition, facilitating the disambiguation of feature combinations responsible for specific activations. We integrate dictionary learning with automated interpretation techniques to address the critical need for interpretability in LLMs, explicitly decomposing and analyzing model features. Specifically, we train sparse autoencoders to extract monosemantic features from polysemantic neurons, thereby reducing the ambiguity in individual neuron behavior. Additionally, we provide a comprehensive unsupervised framework for applying mechanistic interpretability analyses to downstream tasks, enabling a clearer understanding of these models’ internal mechanisms. Our contributions are distinct from prior work in several key aspects: • Decomposition of Advanced LLMs : Unlike existing studies focussing on toy models, we decompose several advanced open-source LLMs, directly comparing their performance on downstream tasks. • Modular Application to Downstream Tasks : We are the first to apply autoencoder-based feature decomposition in a modular manner to complex downstream tasks, demonstrating improved performance. • Ambiguity Resolution in Mathematical Symbols : By decomposing LLMs, we discovered that they exhibit ambiguities when processing certain mathematical symbols, hindering their reasoning abilities. We resolve these ambiguities through query reformulation, improving the performance of these LLMs in mathematical problem-solving. • Enhanced Metaphor Detection : Our method leverages internal feature extraction rather than external tools like POS tagging or knowledge graphs Tian et al. 2024 , providing a more direct and accurate analysis of how LLMs understand metaphors. Overall, our work introduces a novel generalisable framework that advances the interpretability of LLMs while simultaneously enhancing their performance in critical downstream tasks. 2 Related Work Interpretability of LLMs Belinkov and Glass 2019 reviewed interpretability methods in neural language models, highlighting the need to disentangle features for a clearer understanding of LLMs’ internal structures. Techniques like attention visualization, layer-wise relevance propagation, and probing classifiers provide partial insights but fail to fully explain model decision-making Karpathy et al. 2015 ; Gupta and Schütze 2018 ; Conneau et al. 2018 . In contrast, sparse representations and feature decomposition show greater potential for producing human-readable interpretations Faruqui et al. 2015 ; Murdoch et al. 2018 . A key challenge in mechanistic interpretability is polysemanticity, where neurons respond to multiple unrelated inputs, complicating function assignment Olah et al. 2020 . Olah et al. 2020 introduced the concept of circuits in LLMs, demonstrating how analyzing neuron activations and their interactions can clarify model behavior. Prior research suggests that polysemantic neurons arise from the network’s need to efficiently compress features, resulting in fewer neurons than distinct features in the data Elhage et al. 2022 . To address this, researchers applied dictionary learning and sparse feature extraction on single-layer toy models to decompose neuron activations into more interpretable units. Sparse representations offer a promising solution to polysemanticity and superposition, with sparse autoencoders reducing neuron overlap to create more monosemantic feature representations. Cunningham et al. 2023 further explored this approach to extract interpretable features from LLM activations, enabling precise identification of key factors influencing counterfactual behavior, such as decision-making in indirect object identification tasks. Another aspect of interpretability involves applying these techniques to specific downstream tasks, such as mathematical reasoning or metaphor detection, where understanding model behavior is critical to improving performance. Doshi-Velez and Kim 2017 called for more rigorous interpretability studies in these specific contexts, arguing that the demand for transparency is particularly

Read the complete text