Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation

Joanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados · 2024

Read Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation by Joanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15375.txt.

Introduction

Joanne Boisson and co-authors set a task for pulling metaphoric analogies out of literary texts, in a 2024 arXiv paper. Such pulls need high-level thought: the reader must spot two pairs that match in a hidden way. The team built a fresh data set with help from domain experts. Big language models were then tried, fresh from the box, on the task.

Models had to lay out the pairs and name parts the text only hints at. Their scores ran close to human-grade work in the team's eyes. The finds hint that machines may one day mine figures from books at scale, a job once left to hand work.

Catalog record

CREATORJoanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15375

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation Joanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados Abstract Extracting metaphors and analogies from free text requires high-level reasoning abilities such as abstraction and language understanding. Our study focuses on the extraction of the concepts that form metaphoric analogies in literary texts. To this end, we construct a novel dataset in this domain with the help of domain experts. We compare the out-of-the-box ability of recent large language models (LLMs) to structure metaphoric mappings from fragments of texts containing proportional analogies. The models are further evaluated on the generation of implicit elements of the analogy, which are indirectly suggested in the texts and inferred by human readers. The competitive results obtained by LLMs in our experiments are encouraging and open up new avenues such as automatically extracting analogies and metaphors from text instead of investing resources in domain experts to manually label data. 1 Introduction According to Hofstadter and Sander (2013) , analogy is the fuel and fire of thinking because humans dynamically build concepts by analogy. Although it is a core mechanism of the mind, they have proven to be difficult to extract automatically from free text, because they can involve some implicit concepts and can link dissimilar concepts. In some cases, in particular when analogies pair very different concepts, they can also be metaphoric Bowdle and Gentner (2005) . For example, a head and an apple can be mapped in the sentence My head is an apple without a core Sternberg et al. (1993) , and form an analogy that is also a metaphor. The recent progress of transformer-based large language models (LLMs) open a path towards a finer-grain semantic handling of metaphoric analogies in Natural Language Processing. Reasoning by analogy at a human like level is an ability pointed out by Mitchell (2021) as lacking in LLMs. These reasoning capabilities allow humans to learn from few examples, and draw from relevant past experiences to tackle new problems. Analogical thinking is a process of generalization and abstraction. Large pretrained language models have some analogical abilities, they can be prompted successfully for analogical reasoning Yasunaga et al. (2024) , and perform zero-shot analogical reasoning in visual task after converting them into language Hu et al. (2023) . Do they also recognize complex metaphoric analogies in texts? Webb et al. (2023) solve a broad range of different analogies, from geometric patterns to short pieces of text, by prompting the first GPT-3 instructed model Brown et al. (2020) ; Ouyang et al. (2022) . Tong et al. (2024) release new resource and baselines for metaphor understanding tasks. Beyond the field of NLP, in linguistics, psychology or neuroscience, research on metaphors in the last five decades have been growing in importance Peng and Khatin-Zadeh (2023) . With the new NLP state of the art, more attention is now given to analogy processing in Artificial Intelligence 1 1 1 https://analogy-angle.github.io/ , with the idea of bridging the gap towards other disciplines. For example, researchers have attempted comparisons of the information or features on which humans and models rely to solve analogies Stevenson et al. (2023) . To our knowledge, there is currently no evaluation of the ability of LLMs to extract mappings involving more than one single pair of concepts from text. Text T1 T2 S1 S2 i Books are like imprisoned souls till someone takes them down from a shelf and frees them. (S. Butler) books shelf (imprisoned) souls < prison > 1 Money is the mother’s milk of politics . (J. Unruh) money politics (mother’s) milk < baby > 1 An election is coming. Universal peace is declared, and the foxes have a sincere interest in prolonging the lives of the poultry . (G. Eliot) < candidates > < voters > foxes poultry 2 Table 1: Example of metaphors and their extracted analogical frames: the pair of target domain concepts (T1 & T2), the pairs of source domain concepts (S1 & S2) and the number of implicit terms of the analogy (i). Implicit terms appear between brackets (<>). Modifiers of the main concepts of the analogy appear between parenthesis. Analogies are parallels or mappings across concepts. Proportional analogies, which are the focus of this study, are an association between two pairs of concepts such as Answer is to riddle what key is to lock . We tackle the question of metaphorical mapping identification at the lexical level, by extracting the core words involved in an analogy from pieces of text and prompting the models to output the eventual implicit concepts involved. For example, the poem verse Memory, a jar of flies. Spin off the lid. (Seibles, 1955) maps memory to a jar , and flies to an implicit concept that could very well be <recollections> . Given a short text containing a metaphor, the task we introduce is to extract a pair of Noun Phrases (NPs) belonging to the source domain of the metaphor, i.e. expressions that are used metaphorically ( jar and flies in our example) and another pair of NPs belonging to the target domain, i.e. the topic being discussed ( memory and <recollections> , such as flies are to the jar what recollections are to memory ). We view this task as a step towards the extraction and structuring of more complex metaphoric mappings from free texts. Such research direction opens the path toward the creation of large scale knowledge base of metaphoric mappings that could be benefit disciplines such as neuroscience, cognitive science, sociology, anthropology or literature. Elucidating freely expressed complex analogies also has the potential to improve NLP downstream tasks such as Question Answering, Natural Language Inference or Machine Translation Li et al. (2024) . This task can provide greater value to NLP systems than merely identifying words used metaphorically in texts, because more information is extracted when both the source and target domains are modeled. For example, indirect biases Bolukbasi et al. (2016) , which extend beyond gender stereotypes and can sometimes manifest as subtle hate speech, can be more effectively identified and addressed through analogy extraction. Our main contributions are the following. We release a dataset of 204 instances composed of sourced metaphors 2 2 2 Most have at least an author and the Metaphor of Mind instances have a complete bibliographic reference. , labelled for source and target concept pairs. We evaluate and compare recent transformer-based language models on a new task: identifying the elements of 4-term metaphoric analogies in literary English texts. We then manually evaluate the ability of the models to generate relevant concepts when one or more of the four elements of the analogy are implicit. 2 Task motivation and definition Our study takes root in the observation that recent large pre-trained language models often explain complex literary metaphors and similes well, as in the example below, where the correct intentionality behind the statement is also stated There are fascists pretending to be humanitarians like cannibals on a health kick eating only vegetarians. (Roger McGough) OpenAI GPT-3.5 generates this explanation: […] It highlights the hypocrisy and deceitfulness of such behavior, likening it to cannibals who, while ostensibly trying to improve themselves,

Read the complete text