A framework for analyzing concept representations in neural models

Burin Naowarat, Hao Tang, Sharon Goldwater · 2026

Read A framework for analyzing concept representations in neural models by Burin Naowarat, Hao Tang, Sharon Goldwater (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01381.txt.

Introduction

Burin Naowarat, Hao Tang, and Sharon Goldwater offer one frame for studying how neural models store human ideas such as topic, speaker, or sound type. Past work looked at linear concept subspaces from many angles, probing and erasing among them, without a shared test.

Their frame checks two things: containment, whether an idea lives fully inside a subspace and not outside it, and disentanglement, whether it stays apart from other ideas. Tests on text and speech models show the subspace itself may not be unique, and the tool used to estimate it changes both scores. The leading erasure method does well on both axes but fades on new data. In speech models, phone facts sit apart from speaker facts, while speaker facts resist being packed into a small subspace.

Catalog record

CREATORBurin Naowarat, Hao Tang, Sharon Goldwater
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A framework for analyzing concept representations in neural models Burin Naowarat, Hao Tang, Sharon Goldwater Abstract Understanding how neural models represent human-interpretable concepts is challenging. Prior work has explored linear concept subspaces from diverse perspectives, such as probing and concept erasure. We introduce a unified framework to study these subspaces along two axes: containment , which tests if a concept is fully represented in a subspace but not outside it, and disentanglement , which tests for isolation from other concepts. In experiments on both text and speech models, we first highlight that concept subspaces may not be uniquely determined, and discuss the implications for concept subspace analysis. Then, we compare properties of concept subspaces estimated using five estimators, proposed in different communities. We find that (1) the choice of estimator impacts the containment and disentanglement properties; (2) the state-of-the-art concept erasure method, LEACE, performs well on both testing axes, but still struggles to generalize to unseen data; and (3) in HuBERT speech representations, phone information is both contained and disentangled from speaker information, while speaker information is hard to contain in a compact subspace, despite being disentangled from phones. 1 1 1 We release the source code at https://github.com/burin-n/concept_space . Figure 1: An example concept subspace w.r.t. two concepts \mathcal{Y} and \mathcal{Y}^{\prime} . In this example, \mathcal{Y} is for gender classes and \mathcal{Y}^{\prime} is for profession. Gender accuracies on \mathcal{X}_{\mathcal{Y}} and \mathcal{X}_{\mathcal{Y}}^{c} quantify retention and leakage, indicating how well gender information is contained in \mathcal{X}_{\mathcal{Y}} (containment metrics, in orange). Profession error rates on \mathcal{X}_{\mathcal{Y}} and \mathcal{X}_{\mathcal{Y}}^{c} quantify purity and interference, indicating how well profession information is disentangled from \mathcal{X}_{\mathcal{Y}} (disentanglement metrics, in blue). 1 Introduction A long-standing question in cognitive science and AI is how internal representations relate to behaviorally relevant or human-interpretable concepts. Empirical studies have suggested that in deep models of language, many binary distinctions and relations (such as past vs. present tense, or "capital-of") are represented as linear directions in the representation space 27 ; 5 ; 32 ; 22 , and that more generally, multi-class concepts (such as part-of-speech) are captured in low-dimensional subspaces 39 ; 10 ; 41 ; 7 ; 18 ; 21 . Despite known exceptions 12 and limitations 35 , linear models underlie much work in interpreting and controlling deep models of speech and language. Nevertheless, different studies have (often implicitly) made different assumptions about what it means to say a concept is represented in a subspace, and thus how we would evaluate that claim. For example, word analogy and probing studies ( 27 ; 2 ; 39 ; 33 , e.g.) have focused on whether concepts (or the relations defined by them) are decodable from a subspace, where that subspace is implicitly defined by finding the directions that best differentiate the concept labels or exemplars. Meanwhile, concept erasure studies 34 ; 36 ; 4 have focused on whether concepts are disentangled : they aim to explicitly identify a concept subspace that can be removed while retaining as much information as possible about other concepts. As well as focusing on different goals, these and other studies have used a range of different methods to identify the subspace(s) of interest, but with little or no systematic comparison between these methods. In this paper, we formalize a framework for characterizing how concepts are represented, both individually and in relation to each other. This framework clarifies the assumptions behind different lines of work on concept representation, highlights the role of the concept (subspace) estimation method, and leads to four measures that can evaluate different kinds of claims. More specifically, our measures capture two higher-level properties: the degree to which a particular subspace contains all information about a concept, and the degree to which the concept subspace is disentangled from other concepts of interest. In particular, given a concept and a corresponding subspace, we measure containment via retention (how much information about the concept can be decoded from the linear subspace) and leakage (how much information about the concept can be identified outside of the linear subspace). We measure disentanglement via purity (how much information about other concepts is encoded in the linear subspace) and interference (how much information about other concepts is affected by linear manipulation of the concept subspace). Related measures have been proposed for causal interventions on LLMs 21 , but our measures are more widely applicable, and we illustrate them on speech and masked language models. Our experiments on text (BERT) and speech (HuBERT) models illustrate how the proposed framework can shed light on the way concepts are represented and show that conclusions can be affected by the choice of concept estimation method. First, we use gender and profession concepts in BERT to demonstrate that the four measures are complementary and to emphasize that concept subspaces may not be uniquely determined, which has implications for structural analyses of concept representations. Next, we conduct a comprehensive comparison of concept estimators for phone and speaker concepts in HuBERT. The results indicate that (1) LEACE, an estimator from the concept erasure literature 4 , finds concept subspaces with strong overall properties but exhibits leakage on unseen data, despite its no-leakage guarantee on seen data; and (2) while many estimators are able to find contained and disentangled phone subspaces, all fail to find speaker subspaces with good containment when testing with unseen speakers. Lastly, we further examine this speaker containment limitation and show that estimating a speaker subspace that is well-contained, compact, and capable of generalizing to unseen speakers is non-trivial. Overall, our experiments demonstrate the importance of the proposed measures for concept subspace analysis and illustrate how the framework can be used to draw novel insights into concept representations. 2 Concept representations We begin with our definition of a concept and our framework for exploring how concepts are represented. We use \mathcal{X} to denote a D -dimensional representation space (for example, of model activations). We define a concept \mathcal{Y} as the set of possible outcomes of a random variable. In text representations, for example, gender and profession are both concepts with a small discrete set of labels, 2 2 2 Both linguistic gender and social gender are concepts in text; here we refer to social gender, and define it as a binary concept so we can compare to prior work 4 . while in speech representations, phonetic classes and speaker identity are also discrete concepts. Though alternative definitions of concepts are possible 32 ; 4 ; 23 ; 13 , our definition is consistent with considerable prior work, including studies that probe for concepts ( 2 ; 39 , e.g.) , examine their geometric structure 7 ; 18 , seek to remove them from the representation space 34 ; 36 , or intervene to change one outcome to another 21 . We define a concept subspace for the concept \mathcal{Y} as a subspace \mathcal{X}_{\mathcal{Y}}\subseteq\mathcal{X} . The fact that \mathcal{X}_{\mathcal{Y}} is a subspace means that it can be described as a span of basis elements y_{1},\dots,y_{K}\in\mathcal{X} , or \displaystyle\mathcal{X}_{\mathcal{Y}}=\text{span}(\{y_{1},\dots,y_{K}\}). (1) The goal of estimating a concept subspace becomes finding a set of basis elements that spans that space, and we discuss several estimators in Section 3 . 2.1 Containment and disentanglement measures According to these definitions,

Read the complete text