FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression

Chu Duong Huy Phuoc · 2025

Read FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression by Chu Duong Huy Phuoc (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12336.txt.

Introduction

Phuoc Duong Huy Chu reports a system built for the CoMeDi shared task, a contest on modeling disagreement among people who label text. Human judges often differ on meaning, and most systems erase those differences by forcing one gold label. This system instead ranks pairs of judges by how much they disagree.

Sentences were turned into embeddings by a multilingual model, XLM-RoBERTa, which captures meaning across languages. A deep neural regression network then predicted the average gap between two judges' scores for each item. Design choices such as batch normalization and dropout helped the model generalize. The system reached competitive scores on the contest's correlation measure. Chu argues that strong embeddings and direct modeling of judgment gaps are the keys to ranking disagreement well in multilingual settings.

Catalog record

CREATORChu Duong Huy Phuoc
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12336

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression Chu Duong Huy Phuoc Abstract This paper presents results of our system for CoMeDi Shared Task, focusing on Subtask 2: Disagreement Ranking. Our system leverages sentence embeddings generated by the paraphrase-xlm-r-multilingual-v1 model, combined with a deep neural regression model incorporating batch normalization and dropout for improved generalization. By predicting the mean of pairwise judgment differences between annotators, our method explicitly targets disagreement ranking, diverging from traditional "gold label" aggregation approaches. We optimized our system with a customized architecture and training procedure, achieving competitive performance in Spearman correlation against mean disagreement labels. Our results highlight the importance of robust embeddings, effective model architecture, and careful handling of judgment differences for ranking disagreement in multilingual contexts. These findings provide insights into the use of contextualized representations for ordinal judgment tasks and open avenues for further refinement of disagreement prediction models. 1 Introduction The CoMeDi Shared Task Subtask 2: Mean Disagreement Ranking with Ordinal Word-in-Context Judgments (DisWiC) Schlechtweg et al. 2025 focuses on predicting annotator disagreement in semantic similarity judgments. Participants were tasked to rank word-use pairs based on the mean of pairwise absolute differences in annotations, highlighting disagreement rather than consensus. This task builds on recent research emphasizing the importance of capturing variability in linguistic judgments for complex, ambiguous datasets. Evaluations were using Spearman’s correlation. In this paper, we present an embedding-based approach that uses SentenceTransformer (paraphrase-xlm-r-multilingual-v1) with base model is XLM-RoBERTa Conneau et al. 2020 to generate contextual embeddings for word-use pairs. These embeddings were combined in a deep regression model with Batch Normalization, Dropout, and an optimized learning rate scheduler to enhance performance. The model was fine-tuned to predict disagreement scores efficiently, demonstrating the potential of leveraging advanced multilingual embeddings and robust neural architectures for capturing semantic complexities in multilingual datasets. 2 Related Work Annotation disagreements in NLP, particularly in tasks involving meaning in context, pose challenges to data quality and model reliability. Early studies, such as Artstein and Poesio 2008 and Hovy et al. 2013 , explored inter-annotator agreement and aggregation methods to address inconsistencies. Recent works have shifted toward leveraging disagreements as valuable signals. For instance, Basile et al. 2021 introduced perspectivism to embrace diverse annotator viewpoints, while Mostafazadeh Davani et al. 2022 and Mostafazadeh Davani et al. 2022 utilized disagreements to train models better suited for subjective tasks. In Word-in-Context (WiC) tasks, Schlechtweg et al. 2018 proposed the DURel framework to capture semantic relatedness using ordinal scales, with subsequent studies, such as Uma et al. 2021 , focusing on preserving disagreement information through alternative label aggregation methods. This Subtask 2 builds on this foundation by explicitly modeling disagreement using mean pairwise judgment differences, evaluated via Spearman’s correlation Zar 2005 , offering a novel perspective on handling annotation variability. 3 Task Description The CoMeDi shared task, part of the COLING 2025 workshop Schlechtweg et al. 2025 , consists of two subtasks focusing on predicting disagreements in word sense annotation in context (WiC). The first subtask (OGWiC) involves predicting the median of annotator judgments on an ordinal scale (1-4) for word usage pairs, treating this as an ordinal classification task. The second subtask (DisWiC) aims to rank instances based on the mean disagreement between annotators, measured by pairwise absolute differences in judgments. Both subtasks rely on datasets such as the DWUG EN dataset Schlechtweg et al. 2024 and will be evaluated using Krippendorff’s \alpha Krippendorff 2018 for OGWiC and Spearman’s \rho for DisWiC. 3.1 Dataset We conducted our experiments using the dataset provided by the organizers for training and evaluation. The dataset includes samples from seven languages: Chinese Chen et al. 2023 , English Schlechtweg et al. 2024 , German Schlechtweg et al. 2024 , Norwegian Kutuzov et al. 2022 , Russian Rodina and Kutuzov 2020 ; Kurtyigit et al. 2021 , Spanish Zamora-Reina et al. 2022 , and Swedish Schlechtweg et al. 2024 . Tables 1 and 2 summarize its key characteristics. The training dataset contains more samples than the development set, ranging from 1,222 for Norwegian to 24,891 for Russian. On average, context length varies widely, with Spanish having the longest at 84.72 tokens and Chinese the shortest at 1.00 token. German has the largest maximum context length of 1,643 tokens, while Chinese remains the smallest at 1 token. This diversity in sample sizes and context lengths across languages poses challenges for model generalization but provides a strong foundation for evaluating multilingual methods. Languages # Samples Avg. Len. Max Len. Chinese 20.46 1.00 1.00 English 10.83 31.91 176.00 German 13.69 39.39 1643.00 Norwegian 6.04 47.49 346.00 Russian 12.69 24.88 356.00 Spanish 9.33 84.72 480.00 Swedish 9.11 34.89 376.00 Table 1: Training dataset statistics. Languages # Samples Avg. Len. Max Len. Chinese 3.09 1.00 1.00 English 1.90 32.01 169.00 German 2.59 33.52 376.00 Norwegian 871 52.89 452.00 Russian 1,932 23.98 352.00 Spanish 1,269 82.19 493.00 Swedish 1.41 33.66 305.00 Table 2: Development dataset statistics. 4 System Overview Our system tackles the shared task by combining neural sentence embeddings and a deep regression model to predict mean disagreement rankings for the DWUGs dataset Schlechtweg et al. 2024 . The primary steps include: (i) generating semantic representations using multilingual pre-trained models, (ii) concatenating embeddings for context pairs, (iii) training a regression model to predict mean disagreement values. 4.1 Semantic Representations We employ the SentenceTransformer paraphrase-xlm-r-multilingual-v1 model to generate semantic embeddings for sentence pairs. This model is based on XLM-RoBERTa Conneau et al. 2020 , a transformer architecture fine-tuned for multilingual sentence representation tasks. Given a context sentence, C , the embedding function E(C) produces a 768-dimensional vector: E(C)\in\mathbb{R}^{768} For each data sample, two contexts C_{1} and C_{2} are processed, and their embeddings are concatenated: X=[E(C_{1}),E(C_{2})]\in\mathbb{R}^{1536} 4.2 Deep Regression Model We propose a deep feedforward neural network to map concatenated embeddings to mean disagreement scores. The model architecture consists of: Input Layer: 1536-dimensional concatenated embeddings. Hidden Layers: Four fully connected layers with dimensions [512, 256, 128, 64], each followed by BatchNorm and dropout ( p=0.3 ). Output Layer: A single neuron for regression output. Each hidden layer uses ReLU activation, and the loss function is Mean Squared Error (MSE): \mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\left(y_{i}-\hat{y}_{i}\right)^{2} where y_{i} and \hat{y}_{i} are the ground truth and predicted scores. Figure 1: The structure of Deep Regression model. 4.3 XLM-RoBERTa As illustrated in Figure 2 , the structure of XLM-RoBERTa Conneau et al. 2020 consists of three main components: Embedding Layers, Transformer Encoders, and a final layer for handling specific tasks. During the model’s training process, the input is a sequence of tokens, starting with the [CLS] character. The representation of the sequence is extracted from the vector C, corresponding to the [CLS] token. This vector is passed through a Fully Connected Layer and then processed using the sigmoid activation function to convert the output into a probability value. This value is optimized through the cross-entropy loss function. Figure 2: Structure of BERT and XLM-RoBERTa. 4.4 Training Strategy The model is trained using the AdamW optimizer with weight decay and an initial learning rate of 10^{-4} . To prevent overfitting, we employ learning rate scheduling via ReduceLROnPlateau , reducing the learning rate by a factor of 0.5 if the validation loss does not improve for

Read the complete text