Notebook for the CheckThat! Lab at CLEF 2025
Matteo Fasulo, Luca Babboni, Luca Tedeschini · 2025
Read Notebook for the CheckThat! Lab at CLEF 2025 by Matteo Fasulo, Luca Babboni, Luca Tedeschini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11764.txt.
Introduction
Matteo Fasulo, Luca Babboni, and Luca Tedeschini wrote this 2025 notebook paper for the CheckThat! Lab at CLEF 2025. Their team, AI Wizards, entered a task on spotting subjectivity in news writing: sorting sentences into subjective or objective across many languages. Training data came in five languages, while final scoring added unseen ones like Greek and Polish.
Their main trick adds a sentiment score, from a helper model, to the sentence embedding of each sentence before the call is made. They tried the design with three base models and tuned decision thresholds to fix class imbalance. Sentiment clearly lifted scores, above all for catching subjective sentences. The team took first place for Greek in the shared task.
Catalog record
| CREATOR | Matteo Fasulo, Luca Babboni, Luca Tedeschini |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-11764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Notebook for the CheckThat! Lab at CLEF 2025 Matteo Fasulo, Luca Babboni, Luca Tedeschini Abstract This paper presents AI Wizards ’ participation in the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles, classifying sentences as subjective/objective in monolingual, multilingual, and zero-shot settings. Training/development datasets were provided for Arabic, German, English, Italian, and Bulgarian; final evaluation included additional unseen languages (e.g., Greek, Romanian, Polish, Ukrainian) to assess generalization. Our primary strategy enhanced transformer-based classifiers by integrating sentiment scores, derived from an auxiliary model, with sentence representations, aiming to improve upon standard fine-tuning. We explored this sentiment-augmented architecture with mDeBERTaV3-base, ModernBERT-base (English), and Llama3.2-1B. To address class imbalance, prevalent across languages, we employed decision threshold calibration optimized on the development set. Our experiments show sentiment feature integration significantly boosts performance, especially subjective F1 score. This framework led to high rankings, notably 1st for Greek (Macro F1 = 0.51 =~0.51 ). [orcid=0000-0002-7019-3157, [email protected], url=https://github.com/MatteoFasulo, ] *1 1 [orcid=0009-0001-5260-7467, [email protected], url=https://github.com/ElektroDuck, ] 1 [orcid=0009-0006-0375-829X, [email protected], url=https://github.com/LucaTedeschini, ] 1 keywords subjectivity detection,transformers ,multilinguality ,sentiment-based features ,threshold calibration † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2025 Working Notes, 9 – 12 September 2025, Madrid, Spain † † address: Department of Computer Science and Engineering (DISI) - University of Bologna † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Our work addresses subjectivity detection as defined in Task 1 Ruggeri et al. 2025 of the CLEF 2025 CheckThat! Lab. An overview of the CheckThat! Lab and its constituent tasks can be found in Alam et al. 2025a ; Alam et al. 2025b . Specifically, Task 1 challenges systems to classify sentences from news articles as subjective (SUBJ) or objective (OBJ). This capability is vital for efforts to combat misinformation and improve fact-checking, as the ability to separate opinion from factual claims is essential, particularly given the rapid growth of multilingual online content. A key difficulty in this task lies in its sentence-level granularity, requiring classification without the wider context of the full article. Historically, subjectivity detection approaches, from lexicon-based methods to classical machine learning Kamal 2013 , faced limitations with linguistic variety and subtlety. While contemporary transformer-based models offer substantial advancements, their deployment in diverse multilingual and resource-constrained environments continues to pose challenges. This paper presents our system, which fine-tunes transformer-based models by strategically fusing external sentiment information. We augment sentence representations with sentiment scores from an auxiliary model before classification. This sentiment-enhanced strategy is evaluated on: • mDeBERTaV3-base He et al. 2021a ; He et al. 2021b , ModernBERT-base Warner et al. 2024 , and Llama3.2-1B et al. 2024 fine-tuned with and without our sentiment feature fusion for multilingual subjectivity detection. • The systematic integration of sentiment scores as a key feature engineering step, demonstrating its impact on improving subjective content classification. • The application of decision threshold calibration to mitigate class imbalance inherent in the provided datasets, further refining performance. We evaluate our system across monolingual, multilingual, and zero-shot settings, focusing on improving the F1 score for the subjective class. Our work aims to provide insights into effective strategies for multilingual subjectivity detection, highlighting benefits of integrating sentiment features and careful handling of imbalanced data within a transformer-based framework. 2 Related Work Subjectivity detection, often used as a preprocessing step to sentiment analysis Wilson et al. 2005 , aims to filter out objective content and retain subjective sentences, which are then analyzed for polarity. While the two tasks are closely intertwined and can function complementarily Naveed et al. 2022 , this pipeline-based approach has been common in early works. Subjectivity detection initially relied on lexical resources (e.g., SentiWordNet Baccianella et al. 2010 ) and rule-based systems. While interpretable, these methods lacked adaptability to diverse linguistic expressions and contexts. This limitation was partially addressed by machine learning techniques leveraging engineered features (e.g., n-grams, POS tags), which, however, still faced generalization issues. The advent of deep learning, particularly transformer-based models like BERT Devlin et al. 2019 , has significantly advanced NLP tasks, including both subjectivity detection and sentiment classification. These models learn rich contextual representations from large unlabeled corpora, enabling superior performance when fine-tuned. Our work aligns with this literature by combining both perspectives: since our goal is to identify subjective sentences, we leverage sentiment analysis signals to reinforce subjectivity predictions—an approach supported by prior findings that highlight the strong interdependence between subjectivity and sentiment Naveed et al. 2022 . Additionally, previous CLEF CheckThat! Labs have also demonstrated the effectiveness of transformer architectures for related subtasks, such as identifying subjective claims in news articles Leistra and Caselli 2023 . Multilingual subjectivity detection introduces further complexities. While models like mBERT Devlin et al. 2019 or XLM-R Conneau et al. 2020 provide strong cross-lingual transfer baselines, their performance varies across language pairs and task specificities. mDeBERTaV3, with its disentangled attention mechanism He et al. 2021a ; He et al. 2021b , has shown strong performance on NLU benchmarks, making it suitable here. More recent models like ModernBERT Warner et al. 2024 aim for comparable performance with improved efficiency, often focusing on English. Augmenting text representations with auxiliary information, like sentiment or emotion, for improved classification is an active research area. Similar to the use of emotions in sexism detection Muti A. 2023 , we hypothesize that explicit sentiment signals can help disambiguate subjective statements. Addressing class imbalance is another crucial aspect, especially as one class is often more prevalent in real-world datasets. Techniques range from data-level resampling to algorithmic approaches like cost-sensitive learning or threshold adjustment Abdelhamid and Desai 2024 . Our decision threshold calibration aligns with findings that post-hoc adjustments can effectively improve performance on imbalanced datasets without altering the training process. 3 Dataset The data for this task is provided by the CLEF 2025 CheckThat! Lab Task 1 organizers. 1 1 1 CLEF 2025 CheckThat! Lab Task 1 Data The dataset consists of sentences extracted from news articles across five languages: Arabic (AR), Bulgarian (BG), English (EN), German (DE), and Italian (IT). Each sentence is labeled as either subjective (SUBJ) or objective (OBJ). The annotation guidelines, as described in Antici et al. 2024 , define subjective sentences as ”those expressing personal opinions, sarcasm, exhortations, discriminatory language, or rhetorical figures conveying an opinion. Objective sentences include factual statements, reported third-party opinions, open-ended comments, and factual conclusions”. For each language, the data is split into training, development (dev), and development-test (dev-test) sets. An analysis of the label distribution (Table 1 in Section 5 ) reveals a notable class imbalance across all languages, with the objective class being more frequent. Italian and Arabic exhibit the most pronounced imbalance. This characteristic significantly influences model training and evaluation, necessitating strategies to mitigate its impact. Table 1: Distribution of objective (OBJ) and subjective (SUBJ) labels across different languages and dataset splits. The table presents statistics for the training, development (Dev), and development-test (Dev-Test) sets. Language Training Dev Dev-Test OBJ SUBJ OBJ SUBJ OBJ SUBJ Arabic 1,391 1,055 266 201 425 323 Bulgarian 406 323 175 139 143 107 English 532 298 240 222 362 122 German 492 308 317 174 226 111 Italian