Investigating Idiomaticity in Word Representations
Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton ∗, Marco Idiart ∗∗, Aline Villavicencio ∗ · 2024
Read Investigating Idiomaticity in Word Representations by Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton ∗, Marco Idiart ∗∗, Aline Villavicencio ∗ (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02610.txt.
Description
Abstract Idiomatic expressions are an integral part of human languages, often used to express complex ideas in compressed or conventional ways (e.g. eager beaver as a keen and enthusiastic person). However, their interpretations may not be straightforwardly linked to the meanings of their individual components in isolation and this may have an impact for compositional approaches.
Catalog record
| CREATOR | Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton ∗, Marco Idiart ∗∗, Aline Villavicencio ∗ |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02610
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating Idiomaticity in Word Representations Wei He, Tiago Kramer Vieira, Marcos Garcia, Carolina Scarton ∗, Marco Idiart ∗∗, Aline Villavicencio ∗ Abstract Idiomatic expressions are an integral part of human languages, often used to express complex ideas in compressed or conventional ways (e.g. eager beaver as a keen and enthusiastic person). However, their interpretations may not be straightforwardly linked to the meanings of their individual components in isolation and this may have an impact for compositional approaches. In this paper, we investigate to what extent word representation models are able to go beyond compositional word combinations and capture multiword expression idiomaticity and some of the expected properties related to idiomatic meanings. We focus on noun compounds of varying levels of idiomaticity in two languages (English and Portuguese), presenting a dataset of minimal pairs containing human idiomaticity judgments for each noun compound at both type and token levels, their paraphrases and their occurrences in naturalistic and sense-neutral contexts, totalling 32,200 sentences. We propose this set of minimal pairs for evaluating how well a model captures idiomatic meanings, and define a set of fine-grained metrics of Affinity and Scaled Similarity, to determine how sensitive the models are to perturbations that may lead to changes in idiomaticity. Affinity is a comparative measure of the similarity between an experimental item, a target and a potential distractor, and Scaled Similarity incorporates a rescaling factor to magnify the meaningful similarities within the spaces defined by each specific model. The results obtained with a variety of representative and widely used models indicate that, despite superficial indications to the contrary in the form of high similarities, idiomaticity is not yet accurately represented in current models. Moreover, the performance of models with different levels of contextualisation suggests that their ability to capture context is not yet able to go beyond more superficial lexical clues provided by the words and to actually incorporate the relevant semantic clues needed for idiomaticity. By proposing model-agnostic measures for assessing the ability of models to capture idiomaticity, this paper contributes to determining limitations in the handling of non-compositional structures, which is one of the directions that needs to be considered for more natural, accurate and robust language understanding. The source code and additional materials related to this paper are available at our GitHub repository 1 1 1 https://github.com/risehnhew/Finding-Idiomaticity-in-Word-Representations . 12024 \pageonefooter Action editor: Tal Linzen. Submission received: 22 April 2024; revised version received: 29 August 2024; accepted for publication: 22 October 2024. Issue: 1 1 Introduction The evolution of word representation models has resulted in models with seemingly remarkable language abilities. Not surprisingly these models have been found to store a wealth of linguistic information Henderson (2020) ; Manning et al. (2020) ; Vulić et al. (2020) ; Lenci et al. (2022) , displaying high levels of performance on various tasks ranging from the abilities of even the static models of detecting semantic similarities between different words Lin (1999) ; Mikolov et al. (2013) ; Baroni, Dinu, and Kruszewski (2014) to those of contextualised models of grouping representations in clusters which seem to be related to the various senses of the word Schuster et al. (2019) and can be matched to specific sense definitions Chang and Chen (2019) . While substantial evaluation efforts have concentrated on word and subword units and on larger compositional combinations derived from them, there is less understanding about their ability for handling less compositional structures, such as those found on multiword expressions (MWEs), like noun compounds (NCs) Garcia et al. (2021a) , verb-noun combinations King and Cook (2018) ; Hashempour and Villavicencio (2020) and idioms Yu and Ettinger (2020) ; Dankers, Lucas, and Titov (2022) . Indeed, MWEs include a variety of distinct phenomena and have been described as interpretations that cross word boundaries Sag et al. (2002) , whose meanings are not always straightforwardly derivable from the meanings of their individual components. Moreover, although they include, on the one hand, more transparent and compositional expressions (like salt and pepper ) or expressions with implicit relations (like olive oil as oil made from olives), on the other hand they also include more idiomatic expressions (like eager beaver as a person who is willing to work very hard 2 2 2 Definition from the Cambridge dictionary. ), falling into a continuum of idiomaticity 3 3 3 We understand idiomaticity as semantic opacity , and its continuum as different degrees of opacity . Sag et al. (2002) ; Fazly, Cook, and Stevenson (2009) . This leads to potential problems for models if they follow the Principle of Compositionality Frege (1956) ; Montague (1973) , building the meaning of a larger unit (like a sentence or an expression) from a combination of the individual meanings of the words that are contained in it, as this would result in potentially incomplete or incorrect interpretation for more idiomatic cases (e.g. the idiomatic eager beaver interpreted literally as impatient rodent ). Although understanding the meaning of an MWE may require knowledge that goes beyond that of the meanings of these individual words in isolation Nunberg, Sag, and Wasow (1994) , failure to take idiomaticity into account can affect the quality of downstream tasks Sag et al. (2002) ; Constant et al. (2017) ; Cordeiro et al. (2019) such as reasoning and inference Chakrabarty, Choi, and Shwartz (2022) ; Chakrabarty et al. (2022) ; Saakyan et al. (2022) , information retrieval Acosta, Villavicencio, and Moreira (2011) and machine translation Dankers, Lucas, and Titov (2022) . For machine translation, for example, the degree of idiomaticity and ambiguity of MWEs (literal vs. idiomatic usages) were found to have an impact on the quality of the results obtained Dankers, Lucas, and Titov (2022) . Due to their non-compositional nature, idiomatic expressions result in lower quality translations than literal expressions, as evidenced by lower BLEU scores for translations that are paraphrased rather than translated word-for-word. In this paper, we investigate to what extent widely used word representation models are able to capture idiomaticity in MWEs. We focus, in particular, on their initial abilities for representing idiomaticity, looking at noun compounds of varying degrees of idiomaticity. 4 4 4 We use the off-the-shelf publicly available pre-trained versions of widely adopted word representation models, standard operations and common similarity measures. Even in scenarios in which adopting additional optimisations, more complex operations or fine-tuning could lead to improvements in performance, this may depend on the availability of comprehensive training data for the target model, domain and language. Measuring the initial idiomaptic abilities of models can help understand the potential loss of idiomatic meaning that could be propagated to the downstream tasks that use them off-the-shelf. In addition to the complex interactions between MWEs, their component words and their contexts Sag et al. (2002) , characteristics of languages and of word representation models may affect how accurately MWEs can be represented and processed, and we investigate the impact of some of these factors for compounds in two different languages (English and Portuguese). One of the challenges is that uncovering how word representation models capture a specific type of knowledge is a non-trivial problem Vulić et al. (2020) , and may depend on factors like the particular model and the way it encodes different types