Tug-of-war between idiom’s figurative and literal meanings in LLMs

Soyoung Oh, Xinting Huang, Mathis Pink, Michael Hahn, Vera Demberg, Saarland University, Max Planck Institute for Software Systems, Max Planck Institute for Informatics {soyoung, xhuang, mhahn}@lst.uni-saarland.de , [email protected] , [email protected] · 2025

Read Tug-of-war between idiom’s figurative and literal meanings in LLMs by Soyoung Oh, Xinting Huang, Mathis Pink, Michael Hahn, Vera Demberg, Saarland University, Max Planck Institute for Software Systems, Max Planck Institute for Informatics {soyoung, xhuang, mhahn}@lst.uni-saarland.de , [email protected] , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01723.txt.

Introduction

Soyoung Oh and co-authors trace how language models handle idioms, whose figurative meanings clash with their literal ones. Their paper appeared in 2025. The team used causal tracing, switching parts of the model's activity on and off to find what carries each meaning. They located three mechanisms. Early layers fetch the figurative reading and quiet the literal one.

When context comes first, the model uses it from the very first layer and revises its reading if the context clashes. Later, two paths compete: one favors the figurative meaning and a direct route favors the literal one. Both readings stay alive until the end. The authors say this tug-of-war explains both the models' skill with idioms and their occasional literal-minded slips.

Catalog record

CREATORSoyoung Oh, Xinting Huang, Mathis Pink, Michael Hahn, Vera Demberg, Saarland University, Max Planck Institute for Software Systems, Max Planck Institute for Informatics {soyoung, xhuang, mhahn}@lst.uni-saarland.de , [email protected] , [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01723

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tug-of-war between idiom’s figurative and literal meanings in LLMs Soyoung Oh, Xinting Huang, Mathis Pink, Michael Hahn, Vera Demberg, Saarland University, Max Planck Institute for Software Systems, Max Planck Institute for Informatics {soyoung, xhuang, mhahn}@lst.uni-saarland.de , [email protected] , [email protected] Abstract Idioms present a unique challenge for language models due to their non-compositional figurative meanings, which often strongly diverge from the idiom’s literal interpretation. This duality requires a model to learn representing and deciding between the two meanings to interpret an idiom in a figurative sense, or literally. In this paper, we employ tools from mechanistic interpretability to trace how a large pretrained causal transformer (LLama3.2-1B-base) deals with this ambiguity. We localize three steps of idiom processing: First, the idiom’s figurative meaning is retrieved in early attention and MLP sublayers. We identify specific attention heads which boost the figurative meaning of the idiom while suppressing the idiom’s literal interpretation. The model subsequently represents the figurative representation through an intermediate path. Meanwhile, a parallel bypass route forwards literal interpretation, ensuring that a both reading remain available. Overall, our findings provide a mechanistic evidence for idiom comprehension in an autoregressive transformer. 1 1 1 Code and dataset are available at https://github.com/sori424/idiom_processing under the MIT License. $\dagger$ $\dagger$ footnotetext: Co-corresponding authors. 1 Introduction Figure 1: The figurative and literal interpretations are highlighted in the block and path. We find three main steps for idiom processing: 1 Idiom retrieval step: Early layers (i.e., 0-3) attention and MLP are actively retrieving the idiom’s figurative meaning while storing both figurative and literal interpretations in the residual stream. 2 Selective interpretation step: At the token immediately following the idiom span, the model begins to encode a representation that favors the figurative interpretation over the literal one, starting from the middle layers. 3 Interpretation routing: For final prediction, the model passes literal interpretation via both a direct compositional semantic path ( 3-2 ), as well as the intermediate pathway that prioritizes the figurative meaning ( 3-1 figurative path ). Idioms challenge standard semantic composition because they are, by definition, multi-word sequences whose figurative meanings diverge from the composition of their literal parts Beck and Weber 2020 . For instance, “kick the bucket” conveys the figurative meaning “to die”, yet its literal sense “physically kicking a container” remains semantically available. This ambiguity often leads large language models (LLMs) to misinterpret idioms, favoring their literal sense over the intended figurative one Kabra et al. 2023 ; Liu et al. 2023 ; Knietaite et al. 2024 . Psycholinguistics has been investigated how humans understand idioms, whether they access the idiomatic meaning directly or first have to access the literal meaning of the idiom and only in a later stage suppress that meaning and access the figurative meaning. These findings have lead to various models describing human idiom processing Gibbs 1980 ; Cacciari and Glucksberg 1991 ; Bobrow and Bell 1973 ; Cacciari and Tabossi 1988 . Yet for LLMs, it’s unclear how they process idioms, potentially limiting further improvements in their idiom comprehension. One explanation may lie in how the model transforms raw input token embeddings into richer representations Tian et al. 2023 ; Haviv et al. 2022 ; Dankers et al. 2022 . Within the autoregressive transformer, token representations are incrementally refined from the initial input embeddings to the final representation used for next token prediction Vig et al. 2020 . This transformation is achieved through a series of residual layers, which consist of multi-head self-attention (MHSA) and MLP sublayers, each adding their outputs to update the representation. This multi-stage transformation suggests that idiom processing might follow a multi-step transformation process. Haviv et al. 2022 demonstrated that the early layers of transformer models store and retrieve idiomatic information, while the higher layers enhance model confidence. Meanwhile, Dankers et al. 2022 showed that during machine translation of idioms, the transformer model recognizes idiomatic expressions and processes them more as single lexical units. This involves grouping the idioms together using attention sublayers and reducing their interaction with surrounding context. However, both studies lack a detailed mapping of the specific model components and causal pathways underlying these idiom processing behaviors. In this paper, we first aim to identify how the idiom is interpreted by the transformer (i.e., LLama3.2-1B-base), where its figurative meaning is located, and how it is retrieved by the model. To this end, we pinpoint specific components of the model that have specialized in processing idioms by boosting their figurative meaning, while suppressing the compositional semantic literal interpretation. Having identified these, we proceed to investigating the information flow in the model, with the goal of identifying specific pathways through which the figurative meaning is passed forward to the final prediction, and whether this differs from the pathways of passing forward information about an idiom’s literal interpretation. We address these questions through knockout analyses Nanda et al. 2023 ; Wang et al. 2022 ; Geva et al. 2023 , where we separately knock out activations of each component (layer-wise MHSA and MLP, individual attention heads) in the model to observe their importance for retrieving idiom’s figurative meaning. Furthermore, to trace the flow of these interpretations, we employ activation patching experiments Wang et al. 2022 ; Meng et al. 2022 ; Hanna et al. 2023 ; Conmy et al. 2023 ; Stolfo et al. 2023 , in which we ablate one or both interpretations from specific pathways within the model. Our main findings are summarized in Figure 1 : we identify three main steps of idiom processing (1) retrieval of the figurative meaning of the idiom in the early layers, (2) selective encoding of the two interpretations at intermediate layers after the idiom span token position, and (3) two alternative paths for routing the idiom’s meaning to the final prediction, via a direct path for the literal meaning and an intermediate pathway that preferentially amplifies figurative interpretation over the other. In summary, our main contributions are: • We propose a next token prediction task that distinguishes figurative vs. literal interpretations using interpretation-specific tokens (Section 2 ). • We show that early layers contribute to retrieve the idiom’s figurative meaning, and that these are significantly different from general semantic processing (Section 4 ). • We identify a selection mechanism at the token immediately following the idiom, where its residual activation shifts toward the figurative interpretation and then propagates through the intermediate layers (Section 5 ). • We also uncover a bypass pathway that channels the literal interpretation directly into the final prediction, circumventing the intermediate route (Section 5.2 ). 2 Idiom interpretation task We are interested in teasing apart the language model’s processing of the literal vs. figurative interpretation of ambiguous idiom expressions. We therefore formalize a task and construct a dataset to investigate the interpretation shift in the model. In order to allow for effective probing, we embed the idiom into a sentence template, constructed such that the interpretation is revealed via next token prediction. The template 2 2 2 X is instantiated with various pronouns (he/she/it/they); the word “would” is inserted only in some of the idioms to make the sentence sound more fluent; idioms are morphologically fit into the sentence; depending on sg/pl form of X, the second part of the template

Read the complete text