Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders

Mathis Le Bail, Jérémie Dentan, Davide Buscaldi, Sonia Vanier · 2025

Read Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders by Mathis Le Bail, Jérémie Dentan, Davide Buscaldi, Sonia Vanier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23951.txt.

Introduction

Mathis Le Bail and co-authors present in 2025 a model that explains how large language models sort sentences into classes. Tools called sparse autoencoders can pull clear concepts from a model's inner signals. The authors adapt that idea for classification, with a special head for the class task and a loss that keeps concept use sparse.

Their model is tested against several known explanation methods on standard data sets and on varied base models. Two new scores check how precise the concept based explanations are, using a separate sentence encoder. Results show better cause and clarity in the concepts found, which helps readers trust why a model gave a given label.

Catalog record

CREATORMathis Le Bail, Jérémie Dentan, Davide Buscaldi, Sonia Vanier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23951

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders Mathis Le Bail, Jérémie Dentan, Davide Buscaldi, Sonia Vanier Abstract Sparse Autoencoders (SAEs) have been successfully used to probe Large Language Models (LLMs) and extract interpretable concepts from their internal representations. These concepts are linear combinations of neuron activations that correspond to human-interpretable features. In this paper, we investigate the effectiveness of SAE-based explainability approaches for sentence classification, a domain where such methods have not been extensively explored. We present a novel SAE-based architecture tailored for text classification, leveraging a specialized classifier head and incorporating an activation rate sparsity loss. We benchmark this architecture against established methods such as ConceptShap, Independent Component Analysis, and other SAE-based concept extraction techniques. Our evaluation covers two classification benchmarks and four fine-tuned LLMs from the Pythia family. We further enrich our analysis with two novel metrics for measuring the precision of concept-based explanations, using an external sentence encoder. Our empirical results show that our architecture improves both the causality and interpretability of the extracted features. 1 1 1 See code at: https://github.com/orailix/ClassifSAE 1 Introduction Figure 1: 2D Principal Component Analysis (PCA) fitted on sentence-level hidden-state activations extracted from the residual stream of the penultimate transformer block of Pythia-1B Biderman et al. 2023 fine-tuned on AG News Zhang et al. 2015 . The colored circles stand for the concepts learned by our method \verb|ClassifSAE| . Their size is proportional to their mean activation over the dataset. The proportion of color is representative of the normalized class score for each concept. The four triangle depict the class prototypes activations. Text classification, similar to many other NLP tasks, has seen significant performance improvements with the adoption of Large Language Models (LLMs). However, compared to more self-explainable methods, the Transformer architecture used in LLMs does not readily reveal the specific concepts it actually leverages from the text to make a labeling decision. Its relies on a high-dimensional latent space where vectors lack intuitive interpretability. Concept-based interpretability methods aim to extract high-level concepts from this space Poeta et al. 2023 . The latter are directions in the latent space that seek to align with human-understandable ideas, making them more meaningful than the latent vectors themselves. In text classification, the labels may be too simplistic and various notions can result in the same decision. Therefore, extracting more nuanced concepts from the hidden states can provide deeper insights into the model’s intermediate decision-making process. To serve as a good proxy for the concepts learned by a model, the extracted directions should encompass three key properties. Effective concept vectors should be complete and fully reflect the internal mechanism of the inspected neural network. This means that the original layer activations can be reliably reconstructed from the concepts activations. Second, the explanations provided by the activated concepts need to be faithful or causal with respect to the model’s final prediction ( Lyu et al. 2024 ) . The ablation of an identified concept vector must lead to significant variations in the inferred probabilities. Finally, the directions should align closely with well-defined and semantically meaningful human notions. This is measured by the precision and recall of the concepts. A high precision ensures that the activation of a direction is reflected by the presence of a well-defined concept within the input sentence. Conversely, the recall measures how reliably the ground-truth notion activates its associated direction when present in the tested sentence. Since recall depends on ground truth labels, which are unavailable for arbitrary text, we primarily focus on the precision of the extracted concepts. Unsupervised approaches have gained in popularity to construct interpretable linear directions from LLMs latent space. Mechanistic Interpretability is a new field of research that aims to automatically break down complex neural networks into simpler and interpretable parts to gain insights into the overall system Zhao et al. 2024a ; Wang et al. 2022 ; Bills et al. 2023 . Recent contributions in this area built on the superposition linear representation hypothesis Elhage et al. 2022 to design scalable Sparse AutoEncoders (SAEs) for identifying meaningful directions within LLMs latent space without supervision Cunningham et al. 2023 . They are mainly trained in the broad framework of autoregressive prediction, using as input a large number of token embeddings produced by the investigated LLM. A few studies have shown the practicality of SAE features extracted from pre-trained LLMs to obtain high accuracy scores on several text classification datasets Gao et al. 2024 ; Gallifant et al. 2025 . However, to the best of our knowledge, in the context of text classification by language model, no prior research has thoroughly compared SAE-extracted representations and post-hoc methods in the field of concept discovery. In this work, we address this question in the context of autoregressive LLMs fine-tuned for text classification. We propose \verb|ClassifSAE| , a supervised variant of the SAE approach to reveal sentence-level features captured from LLMs’ internal representations. We seek features that are both aligned with interpretable concepts and influential with regard to the classification decisions. From a practical standpoint, we train our SAEs from scratch on the classification dataset, similar to other concept-based reference approaches. This differs from previous methods that pre-train SAEs on larger datasets and select few features that are relevant for the classification task. The method requires a limited number of sentence examples, on the order of 100,000. This allows any user with a model fine-tuned on a given classification task to quickly capture only the concepts related to that setting. Since the expansion dimension of the SAE is larger than the number of relevant concepts for most classification tasks, we enforce the concentration of key concepts within a subset of the hidden layer by training jointly the SAE and a classifier on that subset. Additionally, to prevent the collapse of diversity into a few active features, we design a sparsity mechanism that enables better control over the maximum activation rate of the concepts. We evaluate the quality of the representations by adapting the three key attributes mentioned earlier to this particular setting. We use the accuracy reconstruction to assess completeness . Causality is measured by direct ablations. Finally, we introduce a sentence encoder to assess the precision of the features by quantifying the similarity of sentences that activate the same features. Our paper makes the following contributions: • We propose a new supervised SAE-based method, \verb|ClassifSAE| , to extract fine-grained concepts learned by a LLM trained for sentence classification. We enrich existing SAE interpretability approaches with a joint classifier and an activation rate sparsity mechanism. • We introduce two novel metrics based on an external sentence encoder to assess the precision of sentence-level concepts. • We empirically compare our extracted concepts against two well-established post-hoc techniques for identifying meaningful vectors within the internals of a classifier: ICA Comon 1994 and ConceptSHAP Yeh et al. 2020 , and TopK SAE. 2 Related Work 2.1 Concepts discovery in classification Concept-based explanations provide a more robust understanding of an LLM’s internal processes compared to the often unstable token attributions produced by gradient-based methods Adebayo et al. 2018 . TCAV Kim et al. 2018 was among the first methods to identify vectors in the activation space

Read the complete text