Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings
Hanna Herasimchyk, Alhassan Abdelhalim, Sören Laue, Michaela Regneri · 2025
Read Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings by Hanna Herasimchyk, Alhassan Abdelhalim, Sören Laue, Michaela Regneri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13729.txt.
Introduction
Hanna Herasimchyk and three co-authors warn that good forecasts from word vectors do not prove the vectors hold that knowledge. Past work maps vectors to human trait lists and takes high forecast scores as proof of meaning. They test that claim head on.
The same maps forecast even random trait lists well. Scores thus hit a math ceiling set by the method, not by true sense in the vectors. Set to set ranks by forecast score mislead about which set vectors hold best. Maps show shape match in space more than true word sense, they argue.
Catalog record
| CREATOR | Hanna Herasimchyk, Alhassan Abdelhalim, Sören Laue, Michaela Regneri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings Hanna Herasimchyk, Alhassan Abdelhalim, Sören Laue, Michaela Regneri Abstract Understanding what knowledge is implicitly encoded in deep learning models is essential for improving the interpretability of AI systems. This paper examines common methods to explain the knowledge encoded in word embeddings, which are core elements of large language models (LLMs). These methods typically involve mapping embeddings onto collections of human-interpretable semantic features, known as feature norms. Prior work assumes that accurately predicting these semantic features from the word embeddings implies that the embeddings contain the corresponding knowledge. We challenge this assumption by demonstrating that prediction accuracy alone does not reliably indicate genuine feature-based interpretability. We show that these methods can successfully predict even random information, concluding that the results are predominantly determined by an algorithmic upper bound rather than meaningful semantic representation in the word embeddings. Consequently, comparisons between datasets based solely on prediction performance do not reliably indicate which dataset is better captured by the word embeddings. Our analysis illustrates that such mappings primarily reflect geometric similarity within vector spaces rather than indicating the genuine emergence of semantic properties. † † paperid: 1202 1 Introduction Large Language Models (LLMs) have emerged as a core tool and central research area in natural language processing. While their capabilities are impressive, we still need to understand how they gain such excellent performance apart from using vast amounts of data and parameters. While many algorithms exist for explaining LLMs and their individual components, many of those approaches are complex and not fully understood themselves. Differentiating faithful explanations from misleading correlations is hard, and "Clever Hans" effects ( Shapira et al. 2024 ) also occur for explainability methods. In this paper, we focus on explanation methods for word embeddings, which serve as the input layer for LLMs. A popular approach for explaining which kind of knowledge is implicitly learned when training such embeddings on large amounts of text is called "property inference" Rosenfeld and Erk 2023 . Property inference aims to detect knowledge (the "properties") about a word implicitly stored in its embedding vector by predicting ("inferring") the properties in question from the embedding. To achieve this, a predictive model is trained to map an embedding vector onto a corresponding set of properties, often taken from curated datasets known as feature norms. These norms provide interpretable, human-annotated semantic features that vary in their relationship to their associated words. Several studies used such norms and property inference to demonstrate that emergent knowledge in embeddings contains semantic, perceptual, or neurological features ( Abnar et al. 2018 ; Chersoni et al. 2021 ; Chronis et al. 2023 ; Fagarasan et al. 2015 ; Hollenstein et al. 2019 ; Nair et al. 2020 ) . The common assumption underlying these studies is that if a model can accurately predict semantic properties from an embedding, then the embedding must encode that knowledge. However, this assumption has not been rigorously validated, and the explanatory capacity of such methods remains uncertain. In this work, we want to validate and differentiate the explanatory potential of property inference methods. To do so, we apply two methods commonly used in previous studies (Partial Least Squares Regression and Feed Forward Neural Networks) to map word embeddings from BERT Devlin et al. 2019 to three different feature norms (two categorical norms and one with continuous features). We extend previous studies by providing multiple ablation experiments and upper bounds, to put the resulting numbers into perspective. Our results show that while information overlap between feature norms and embeddings is predictable, the results are not faithful explanations of emergent properties, but rather a different type of geometric similarity. Significant predictive performance can even be achieved for nonsensical or random features, calling into question whether such methods faithfully explain latent knowledge in embeddings. Our main contributions are as follows: • We show that common property inference methods do not provide faithful explainability. While high information overlap does lead to good results, good results do not indicate high information overlap. • We provide a detailed set of experiments to assess the explanatory potential of property inference methods. • We show that mapping embeddings measures geometric similarity, and we distinguish this from explaining property knowledge. After revisiting previous work (Sec. 2 ), we introduce the methods we use (Sec. 3 ) and our experimental setup (Sec. 4 ). In several ablation experiments, we show that the methods have a low upper bound, which determines more of the results than the information overlap between embeddings and norms (Sec. 5 ). Our last argument shows that the prediction methods explain geometric similarity but not individual properties (Sec. 6 ). we conclude with a discussion of the results (Sec. 7 ), followed by a summary and ideas for future work (Sec. 8 ). Code and data Herasimchyk et al. 2025 as well as additional evaluation results (in the Appendix) are provided as supplementary material. 2 Background and related work Research on embedding explainability was done long before LLMs existed, starting with the first distributional models that derived word meaning from co-occurrence frequencies ( Lund and Burgess 1996 ) . Originally, those models had directly interpretable dimensions because they denoted context words as features. More advanced models like Word2Vec ( Mikolov et al. 2013 ) or GloVe Pennington et al. 2014 used occurrence probabilities rather than frequencies and condensed the vector spaces to fewer dimensions, so the resulting embedding dimensions were not directly interpretable anymore. Embeddings of this type excelled in Natural Language Processing tasks like word sense discrimination ( Schütze 1998 ) or text classification ( Glavaś et al. 2019 ) . At the same time, such embeddings have also been applied to tasks related to cognitive science, like text comprehension ( Peter W. Foltz and Landauer 1998 ) . Sets of cognitively plausible features were originally considered an alternative to distributional embeddings ( Church and Hanks 1989 ) but now serve as a gold standard for potentially emergent knowledge in generative models. Property inference is one such approach for interpretation that maps curated sets of interpretable properties to embeddings. This explainability method has been used to decode semantic features ( Chersoni et al. 2021 ; Chronis et al. 2023 ; Nair et al. 2020 ) , perceptual associations ( Fagarasan et al. 2015 ) , and even neurological patterns from fMRI and EEG data ( Abnar et al. 2018 ; Hollenstein et al. 2019 ) . Most often, partial least squares regression (PLSR) ( Rosipal and Krämer 2006 ) or feedforward neural networks (FFNNs) are used for the mapping, but other approaches have been tested ( Rosenfeld and Erk 2023 provide an overview). As a linear mapping method, PLSR has also found widespread use in fields such as medicine ( Palermo et al. 2009 ) , chemistry ( Wold et al. 2001 ) , or economics ( Dirsehan and Henseler 2023 ) due to its robustness in handling high-dimensional and collinear data. A common problem with taking the resulting predictions as causal interpretations is that it is difficult to dissect whether the results are, as expected, due to information overlap, whether they stem from the analyzed information source, or are correlations with