Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss

Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen · 2026

Read Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss by Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22799.txt.

Introduction

Matheny, Nguyen, and Nguyen improve detection of idioms and figurative language. Idioms carry one meaning that is not the sum of their words. Tokenization makes such phrases hard for language models to spot. The authors fine-tune BERT and RoBERTa models with two losses. A slot loss pairs with a span contrastive loss that reweights hard negatives. The models reach state-of-the-art sequence accuracy on existing datasets. Tests support the contrastive loss and its wider use. They also propose a combined score for span awareness and overall performance.

Catalog record

CREATORBlake Matheny, Phuong Minh Nguyen, Minh Le Nguyen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44696
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22799

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Span Modeling for Idiomaticity and Figurative Language Detection with Span Contrastive Loss Blake Matheny, Phuong Minh Nguyen, Minh Le Nguyen Abstract The category of figurative language contains many varieties, some of which are non-compositional in nature. This type of phrase or multi-word expression (MWE) includes idioms, which represent a single meaning that does not consist of the sum of its words. For language models, this presents a unique problem due to tokenization and adjacent contextual embeddings. Many large language models have overcome this issue with large phrase vocabulary, though immediate recognition frequently fails without one- or few-shot prompting or instruction finetuning. The best results have been achieved with BERT-based or LSTM finetuning approaches. The model in this paper contains one such variety. We propose BERT- and RoBERTa-based models finetuned with a combination of slot loss and span contrastive loss (SCL) with hard negative reweighting to improve idiomaticity detection, attaining state of the art sequence accuracy performance on existing datasets. Comparative ablation studies show the effectiveness of SCL and its generalizability. The geometric mean of F1 and sequence accuracy (SA) is also proposed to assess a model’s span awareness and general performance together. keywords idiom, idiomaticity detection, data annotation, language models 1 Introduction Idiomaticity detection remains a crucial task for language models in natural language understanding (NLU). Figurative language, like idioms, frequently evades detection by even the largest models. Idioms comprise a subset of figurative language representing a single semantic unit in multiple words, which can be compositional or non-compositional. Unless a language model contains these as multi-word expressions explicitly in their pretraining and vocabulary, even a very large language model may fail the detection task. Our work builds upon prior research and datasets developed for the task of idiomaticity and figurative language detection 16 . Concerted efforts using contrastive loss have focused on similarity objectives with a margin as a tunable hyperparameter 9 . This line of research demonstrated improvements over existing methods in terms of F1 score, although the benefits of hard negative mining remained limited. With the introduction of margin adjustment in the similarity-based loss, performance further improved. Training focused on anchor-negative and positive-negative pairs, equating anchors and positives. This effectively doubled the number of training samples by swapping positives and anchors. Subsequent testing evaluated model predictions against human similarity judgments using Semantic Textual Similarity (STS) data from SemEval2022 15 . While our approach follows a similar contrastive learning structure, we focus on span representation rather than sentence representations and pair-wise comparisons. To address these challenges in slot labeling, span contrastive loss (SCL) with weighted hard negatives was developed and combined with slot loss to train BERT- and RoBERTa-based models on both new and widely used idiom and figurative language datasets, including MAGPIE-idioms 7 and P IFL-OSCAR 16 . This research focused not only on objective functions and loss combinations, but also on similarity measurements of target span representations, computed as the average of word embeddings within each span. Our contributions are: 1. Span contrastive loss employed for multi-word logic 2. Cross evaluation on multi-word expresssion (MWE) and figurative language datasets to examine generalizability on diverse datasets seen quantitatively in tables and qualitatively in figures 3. Proposing geometric mean of F1 and sequence accuracy as span-aware performance 4. State of the art performance for sequence accuracy on the MAGPIE dataset (evaluation under random split setting) 2 Related Work This work represents a combination of techniques, building on adaptations of existing architectures and a novel application of joint loss. As such, the relevant literature can be organized into four principal components: (1) Contextual sequence encoders for span and slot representations, (2) Contrastive representation learning and supervised contrastive objectives, (3) Hard negatives and adaptive negative weighting strategies, and (4) Figurative language and idiom understanding (task context). Contextual sequence encoders for span and slot representations Modern natural language understanding (NLU) systems rely heavily on contextual sequence encoders, with Transformer architectures providing a general-purpose mechanism for producing context-sensitive token representations 24 . BERT-style pretraining further improves slot labeling by supplying bidirectional contextual features that can be fine-tuned for task-specific objectives 6 . For structured sequence labeling, globally normalized decoding with conditional random fields (CRFs) remains a standard technique for enforcing label consistency, particularly in NER-like span prediction settings 13 ; 12 . In parallel, phrase- and span-centric encoders such as PhraseBERT demonstrate that explicitly modeling multi-token phrases yields improved semantic representations for span-level tasks, suggesting that phrase-aware objectives can enhance token-level encoders 25 . Our approach aligns with this direction by deriving span representations from contextual token embeddings and directly supervising them through both classification and contrastive losses. Contrastive representation learning and supervised contrastive objectives Contrastive learning has emerged as a powerful framework for shaping representation geometry by pulling semantically related instances together and pushing unrelated instances apart, often using temperature-scaled softmax objectives such as InfoNCE 23 . Large-batch and in-batch negative formulations have shown that contrastive objectives can scale effectively while producing robust representations 4 . In supervised settings, contrastive learning (SupCon) generalizes this framework by defining positive pairs with shared labels and negatives with differing labels, encouraging class-conditional clustering that benefits classification 11 . The span-level contrastive component of our method follows this supervised formulation: span embeddings derived from contextual encoders are contrasted using gold slot labels, complementing the standard token-level cross-entropy (and optional CRF) objectives. Hard negatives and adaptive negative weighting strategies A consistent finding across metric and contrastive learning is that negative sampling strongly influences optimization. Metric learning objectives such as the N-pair loss 22 and triplet loss variants highlight the importance of informative negatives. More recent work, mentioned previously, proposes adaptive contrastive triplet losses, which dynamically adjust margins or weights based on similarity or difficulty, improving convergence while reducing sensitivity to poorly chosen negatives 9 . Analyses of contrastive optimization further show that hard negatives–those close to the anchor in representation space–provide stronger learning signals but may introduce instability or false negatives if used indiscriminately 19 ; 20 . As a result, several methods emphasize adaptive weighting or curriculum-style treatment of negatives rather than binary selection 10 . Our approach follows this direction by identifying confusable negative spans via in-batch similarity and increasing their contribution through reweighting, preserving global separation signals while focusing gradient pressure on close cases. Figurative language and idiom understanding Although our modeling choices are general-purpose, they connect naturally to prior work on figurative language understanding, where surface overlap and contextual ambiguity are common. Recent studies on idiom identification and non-compositional phrase interpretation demonstrate both the strengths and limitations of pretrained contextual encoders, motivating the need for additional inductive bias or training signals beyond standard fine-tuning 2 ; 27 . Metaphor detection and figurative language benchmarks consistently demonstrate that contextualized language representations substantially improve performance over static or feature-based approaches, particularly in capturing semantic dependencies across sentence contexts 8 ; 14 . However, prior work also highlights that figurative language tasks remain challenging due to the high degree of semantic overlap between figurative and literal usages, leading to confusable representations under standard training objectives 26 . These findings suggest that while contextual encoders provide a strong foundation, they benefit from task-specific learning signals that promote semantic separability more directly. From this perspective, span-level supervised

Read the complete text