The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect
Cosimo Iaia, Bhavin Choksi, Emily Wiebers, Gemma Roig, Christian J. Fiebach · 2025
Read The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect by Cosimo Iaia, Bhavin Choksi, Emily Wiebers, Gemma Roig, Christian J. Fiebach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15682.txt.
Introduction
Cosimo Iaia and colleagues show that the match between human and language model word meanings is driven in large part by concreteness. Nouns name either concrete things, like a table, or abstract ideas, like justice. Past work in psychology has long shown that concreteness shapes how words are handled. How language models mark that split was largely unknown.
The team gauged the gaps in meaning that humans use, from judgments on abstract and concrete nouns picked with care. A similarity study showed that the human space aligns with the spaces of language models. Both spaces also align with concreteness ratings the people gave. Tests that removed parts then showed the human-model match is driven by concreteness. It is not driven by other known word traits. A basic trait of meaning thus shapes model spaces.
Catalog record
| CREATOR | Cosimo Iaia, Bhavin Choksi, Emily Wiebers, Gemma Roig, Christian J. Fiebach |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15682
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect Cosimo Iaia, Bhavin Choksi, Emily Wiebers, Gemma Roig, Christian J. Fiebach Abstract The nouns of our language refer to either concrete entities (like a table) or abstract concepts (like justice or love), and cognitive psychology has established that concreteness influences how words are processed. Accordingly, understanding how concreteness is represented in our mind and brain is a central question in psychology, neuroscience, and computational linguistics. While the advent of powerful language models has allowed for quantitative inquiries into the nature of semantic representations, it remains largely underexplored how they represent concreteness. Here, we used behavioral judgments to estimate semantic distances implicitly used by humans, for a set of carefully selected abstract and concrete nouns. Using Representational Similarity Analysis, we find that the implicit representational space of participants and the semantic representations of language models are significantly aligned. We also find that both representational spaces are implicitly aligned to an explicit representation of concreteness, which was obtained from our participants using an additional concreteness rating task. Importantly, using ablation experiments, we demonstrate that the human-to-model alignment is substantially driven by concreteness, but not by other important word characteristics established in psycholinguistics. These results indicate that humans and language models converge on the concreteness dimension, but not on other dimensions. 1 Introduction Figure 1: Schematic of the approach : Words are sampled from the semantic space (top left) and used for an odd-one-out task. The English translations of the German words are Körper : Body, Gesicht : Face, Figur : Figure. For a set of 40 words, we collected a total of 9880 odd-one-out choices. These were converted into a representational dissimilarity matrix (RDM) reflecting pair-wise semantic distances between words. For each language model, a similar computational RDM is created using the word embeddings for the 40 words. The two RDMs are then compared with each other. The study of concreteness has a long history in Psychology and Neuroscience Paivio et al. 1968 . Concreteness, usually measured through a rating task ( Kanske and Kotz 2010 ; Brysbaert et al. 2014 , e.g.,) , refers to the extent to which a concept is related to sensory experience Reilly et al. 2024 . Psycholinguistic research has established that whether words refer to concrete or abstract concepts influences word recognition behavior, indicating that concrete and abstract words may be represented differently Solovyev 2020 . Similarly, human neuroimaging studies have shown consistent differences in brain activation patterns elicited during the processing of concrete and abstract words Fiebach and Friederici 2004 ; Bucur and Papagno 2021 . Altogether, these and similar results strongly indicate that concreteness is an important dimension that critically influences how conceptual semantic knowledge is represented in the human mind and brain. The recent advancements in language models have also led to an increased interest in how they learn and represent concreteness. Without explicitly having learned about the concreteness dimension, language models can predict concreteness ratings with human-like degrees of accuracy Köper and Im Walde 2016 ; Martínez et al. 2025 ; Wartena 2024 . Language models can predict concreteness even cross-linguistically Thompson and Lupyan 2018 , further supporting the universality of this property. However, the question of exactly how similarly language models and humans represent concreteness remains unanswered. In this paper we address this question by measuring the alignment between the word representational spaces of humans and various language models. Using the method of Representational Similarity Analysis (see Section 3.5 below) we ask three fundamental questions: i) Is there a representational alignment between language models and humans in mental representations of single word meaning? ii) Do humans and language models implicitly represent concreteness? iii) Can concreteness independently explain the degree of agreement between the two representational spaces, i.e., is there a concreteness effect on the representational alignment? To address these questions, we first ran an odd-one-out task where we collected behavioral ratings for 9880 German word triplets to derive a representational space of word meanings and their similarities Hebart et al. 2020 ; Turini and Võ 2022 . Participants (N = 40) were asked to determine which of the three words is least similar to the others (i.e., to determine the odd one). Given a triplet of items i, j, k , the odd-one-out task allowed us to retrieve pairwise similarities between two items (for example, j,k ) compared to the same context item ( i ), building an implicitly derived representational space Hebart et al. 2020 . Importantly, word concreteness was neither explicitly probed nor in any other way relevant for this first behavioral task completed by the participants. However, words were selected such that they were semantically similar but varied in concreteness, as will be described in more detail in Section 3 . After the odd-one-out task, we asked participants to rate the same words used in the previous task for the abstract- vs. concreteness, to (i) validate the concreteness estimates used for designing the experiment, (ii) get a more accurate estimate of the explicit concreteness representation for our participants, and (iii) build an explicit representational space for concreteness. We then obtained the representations (or word embeddings) of the same words from popular language models and compared them to the behavioral data. We first show that there is a significant alignment between the representational space derived from human behavioral judgments and from the different language models. Additionally, both systems (human and language model representational spaces) proved to be independently aligned to a representational space derived from concreteness ratings, which can be considered a ‘gold standard’ in psychology ( Brysbaert et al. 2014 , e.g.,) . Finally, to determine the extent to which concreteness drives the alignment, we first measured the partial correlations between the behavioral and model representations using concreteness as a control variable. Following Oota et al. 2024 , we also removed the maximum variance linearly explained by concreteness from the word embeddings. We then used these concreteness-ablated word embeddings to measure changes in the alignment with the behavioral ratings. Our results revealed that the alignment between the language models and humans is critically driven by concreteness. We publicly release our code and resources to replicate our findings. 1 1 1 Code will be available upon publication at https://github.com/bhavinc/concreteness-in-LLMs 2 Related work The concreteness dimension has been shown to be relevant for the organization of word meaning and to influence semantic processing both behaviorally and neurally Huang and Federmeier 2015 ; Montefinese 2019 ; Bucur and Papagno 2021 . For example, concreteness affects memory performance ( Fliessbach et al. 2006 , concrete words are remembered better than abstract words,) and reaction times in lexical decision tasks ( James 1975 , concrete words are processed faster,) . In a similar way, neuroimaging studies have shown different patterns of activation in the brain for abstract words in (mostly left-lateralized) and concrete words (bilaterally) Binder et al. 2005 . These, and related findings, are usually referred to as the concreteness effect ( Bucur and Papagno 2021 ; Solovyev 2020 ; Löhr 2024 , cf.) . Given the importance of concreteness for word meaning processing, how language models represent this dimension has also been a major interest in