Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework
Laura Kopf Nils Feldhus Kirill Bykov Philine Lou Bommer, [-4pt] Anna Hedström Marina M.-C. Höhne Oliver Eberle, [-4pt] Fraunhofer Heinrich-Hertz-Institute, Germany University of Potsdam, Germany {kopf,feldhus,oliver.eberle}@tu-berlin.de [-4pt] {kbykov,pbommer,ahedstroem,mhoehne}@atb-potsdam.de · 2025
Read Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework by Laura Kopf Nils Feldhus Kirill Bykov Philine Lou Bommer, [-4pt] Anna Hedström Marina M.-C. Höhne Oliver Eberle, [-4pt] Fraunhofer Heinrich-Hertz-Institute, Germany University of Potsdam, Germany {kopf,feldhus,oliver.eberle}@tu-berlin.de [-4pt] {kbykov,pbommer,ahedstroem,mhoehne}@atb-potsdam.de (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15538.txt.
Introduction
Laura Kopf and six co-authors present PRISM, a framework for describing what individual neurons in language models encode, in this 2025 paper. Automated tools already write such descriptions, but they tend to be fragile and assume each neuron holds exactly one concept. Evidence increasingly shows single neurons can hold several concepts at once, a trait called polysemanticity.
PRISM is built to capture that mix instead of flattening it. For each feature it produces a richer description that can name multiple concepts where they exist. It also scores the accuracy of the whole description and how well the separate concepts were caught. In broad benchmarks against existing methods, PRISM's descriptions scored as more accurate and more faithful. The authors offer the framework as a step toward reading model internals in a way that matches how tangled they really are.
Catalog record
| CREATOR | Laura Kopf Nils Feldhus Kirill Bykov Philine Lou Bommer, [-4pt] Anna Hedström Marina M.-C. Höhne Oliver Eberle, [-4pt] Fraunhofer Heinrich-Hertz-Institute, Germany University of Potsdam, Germany {kopf,feldhus,oliver.eberle}@tu-berlin.de [-4pt] {kbykov,pbommer,ahedstroem,mhoehne}@atb-potsdam.de |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15538
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework Laura Kopf Nils Feldhus Kirill Bykov Philine Lou Bommer, [-4pt] Anna Hedström Marina M.-C. Höhne Oliver Eberle, [-4pt] Fraunhofer Heinrich-Hertz-Institute, Germany University of Potsdam, Germany {kopf,feldhus,oliver.eberle}@tu-berlin.de [-4pt] {kbykov,pbommer,ahedstroem,mhoehne}@atb-potsdam.de Abstract Automated interpretability research aims to identify concepts encoded in neural network features to enhance human understanding of model behavior. Current feature description methods face two critical challenges: limited robustness and the flawed assumption that each neuron encodes only a single concept (monosemanticity), despite growing evidence that neurons are often polysemantic. This assumption restricts the expressiveness of feature descriptions and limits their ability to capture the full range of behaviors encoded in model internals. To address this, we introduce Polysemantic FeatuRe Identification and Scoring Method ( PRISM ), a novel framework that captures the inherent complexity of neural network features. Unlike prior approaches that assign a single description per feature, PRISM provides more nuanced descriptions for both polysemantic and monosemantic features. We apply PRISM to language models and, through extensive benchmarking against existing methods, demonstrate that our approach produces more accurate and faithful feature descriptions, improving both overall description quality (via a description score) and the ability to capture distinct concepts when polysemanticity is present (via a polysemanticity score). Introduction Large Language Models (LLMs) have rapidly become integral to a range of real-world applications, from software development ( Li et al. 2022 ) to medical diagnostics ( Tu et al. 2025 ) . Despite their growing influence, the internal decision-making processes of these models remain largely opaque. A growing number of approaches aim to understand these black-box systems by analyzing their internal structure in human-interpretable ways, such as mechanistic interpretability ( Wang et al. 2023 ; Nanda et al. 2023 ; Olah, Chris et al. 2020 ) , structured explanations ( Geiger et al. 2022 ; Schnake et al. 2022 ) , advanced feature attributions ( Abnar & Zuidema 2020 ; Ali et al. 2022 ) , and free-text explanations ( Camburu et al. 2018 ; Madsen et al. 2024 ) . A central goal of this research is to assign interpretable, functional roles to individual components such as neurons or attention heads ( Bykov et al. 2023a ; Bau et al. 2017 ; Mu & Andreas 2020 ) . The presence of polysemanticity , the tendency of individual features to encode multiple, semantically distinct concepts or patterns ( Scherlis et al. 2022 ; Elhage et al. 2022 ) , complicates the process of explaining model components, as it defies the common assumption that a single neuron is associated with a single function or pattern. From a coding-theoretic perspective, this reflects how neural capacity is distributed across multiple tasks. While several concept extraction techniques like sparse autoencoders (SAEs) ( Ng 2011 ; Bricken et al. 2023 ) aim to disentangle polysemantic features, many resulting descriptions still reflect multiple concepts ( OpenAI 2023 ) , limiting interpretability. Existing methods provide a single description per feature ( Bills et al. 2023 ; Gao et al. 2024 ; Choi et al. 2024 ; Yu et al. 2024 ; Gur-Arieh et al. 2025 ) , which can obscure the true complexity of a feature’s function. To offer more comprehensive and nuanced feature descriptions, we introduce PRISM , a framework for generating and evaluating multi-concept feature descriptions that considers multiple patterns per feature. In Figure 1 , we present a neuron previously labeled as “possibly monosemantic” ( OpenAI 2023 ) that PRISM reveals to activate for a highly diverse and heterogeneous set of concepts characterized by textual descriptions. Throughout this work, a concept refers to a cluster of token-level inputs that elicit similar contextualized neuron activations, allowing each neuron to be associated with multiple concepts that reflect the diversity of inputs it responds to. Our contributions include: (1) A framework, PRISM , that generates multi-concept descriptions of features (Section 3 ), providing greater precision and granularity compared to existing methods. (2) A set of quantitative evaluations (Section 3.2 ) for comparing multi-concept textual descriptions across different feature description methods. (3) A first multi-concept feature description analysis of language model features, revealing that individual features encode a highly diverse and heterogeneous set of concepts (Section 5 ). By addressing the limitations of single-concept feature descriptions, PRISM enables a systematic and nuanced understanding of internal representations, crucial to advance the interpretability of language models. Our code is made publicly available to the community. 1 1 1 https://github.com/lkopf/prism Figure 1 : Overview of the PRISM framework. PRISM captures multiple concepts per feature, enabling the detection of both polysemantic and monosemantic features, unlike prior approaches that constrain each feature to a single description. For example, feature 3815 in layer 47 was previously labeled as monosemantic ( OpenAI 2023 ) , whereas PRISM reveals that it responds to multiple distinct concepts. Polysemanticity scoring summarizes how diverse the concepts associated with a feature are, while description scoring assesses how well each concept aligns with the feature’s activation distribution. Related Work Automated Interpretability Recent generative AI systems have grown increasingly complex, motivating efforts to scale interpretability. In addition to methods that provide local explanations for individual predictions, a complementary line of research focuses on fully automated descriptions of model components like neurons and their associated functions. In this context, an automated interpretability approach was introduced to generate a textual description for each neuron in GPT-2 XL ( Bills et al. 2023 ) . This method has since become a foundation for subsequent work on feature descriptions ( Cunningham et al. 2024 ; Bricken et al. 2023 ; Choi et al. 2024 ; Gao et al. 2024 ; He et al. 2024 ; Gur-Arieh et al. 2025 ) . Feature Description Evaluation Automated feature description methods often rely on simulation-based approaches, where a model predicts neuron activations for given inputs ( Bills et al. 2023 ) . This approach, widely adopted in recent work ( Cunningham et al. 2024 ; Bricken et al. 2023 ; Choi et al. 2024 ) , evaluates descriptions based on the correlation between simulated and actual activations. Other evaluations include binary classification of activating and non-activating contexts ( Paulo et al. 2024 ) , contrastive methods using distractor samples ( 1 ; Lindsey et al. 2024 ) , and combined approaches that assess input capture accuracy and feature characterization precision ( Gur-Arieh et al. 2025 ; Huang et al. 2023 ; Templeton et al. 2024 ) . Simulation-based approaches herein rely on an external model’s ability to accurately predict neuron activations given a feature extracted by an additional explainer model. Currently, it remains unclear how well LLMs reliably predict activations of other models, adding uncertainty and complexity to the evaluation of feature descriptions. Our approach instead directly compares feature activation distributions to controls, using both parametric and non-parametric evaluation measures. Clustering and Topic Analysis in Text Recent work in topic modeling has relied on unsupervised clustering of language model embeddings to discover latent topics without relying directly on keywords ( Aharoni & Goldberg 2020 ; Grootendorst 2022 ) . While methods like LLooM ( Lam et al. 2024 ) and goal-driven explainable clustering ( Wang et al. 2023a ) use LLMs to extract or assign high-level textual summaries, our work focuses