Notebook for the CheckThat! Lab at CLEF 2025

Tarık Saraç, Selin Mergen, Mucahid Kutlu · 2025

Read Notebook for the CheckThat! Lab at CLEF 2025 by Tarık Saraç, Selin Mergen, Mucahid Kutlu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08265.txt.

Introduction

Tarık Saraç, Selin Mergen, and Mucahid Kutlu stage debates among chatbots to spot science talk on social media in a 2025 paper. The task, from a shared competition, asks whether a post makes a scientific claim, cites a study, or names scientific things at all.

Their council method seats several models as experts under a chair, deliberating to consensus. Solo and team debate formats were tried and beaten by the council on practice data. In the official run, the council placed first at spotting references to studies, though mid-pack on claims and entities. The work tests structured argument as a fact-checking instrument, with honest scores.

Catalog record

CREATORTarık Saraç, Selin Mergen, Mucahid Kutlu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08265

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Notebook for the CheckThat! Lab at CLEF 2025 Tarık Saraç, Selin Mergen, Mucahid Kutlu Abstract In this paper, we present our work developed for the scientific web discourse detection task (Task 4a) of CheckThat! 2025. We propose a novel council debate method that simulates structured academic discussions among multiple large language models (LLMs) to identify whether a given tweet contains (i) a scientific claim, (ii) a reference to a scientific study, or (iii) mentions of scientific entities. We explore three debating methods: i) single debate, where two LLMs argue for opposing positions while a third acts as a judge; ii) team debate, in which multiple models collaborate within each side of the debate; and iii) council debate, where multiple expert models deliberate together to reach a consensus, moderated by a chairperson model. We choose council debate as our primary model as it outperforms others in the development test set. Although our proposed method did not rank highly for identifying scientific claims (8th out of 10) or mentions of scientific entities (9th out of 10), it ranked first in detecting references to scientific studies. [orcid=0009-0004-7598-5825, [email protected] ] 1 [orcid=0009-0002-3284-9490, [email protected] ] 1 *1 [orcid=0000-0002-5660-4992, [email protected] ] *1 keywords Scientific Discourse Detection ,Debating Method ,Large Language Models † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2025 Working Notes, 9 – 12 September 2025, Madrid, Spain † † address: TOBB University of Economics and Technology, Ankara, Türkiye † † address: Qatar University, Doha, Qatar † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction In this work, we present our approach for Subtask 4a (Scientific Web Discourse Detection) Hafid et al. 2025 of the CheckThat! 2025 shared task Alam et al. 2025 . The subtask involves three binary classification problems for a given tweet: • Category 1: Does the tweet contain a scientific claim? • Category 2: Does the tweet reference a scientific study or publication? • Category 3: Does the tweet mention scientific entities, such as a university or scientist? In this work, we propose a few-shot classification approach in which LLMs engage in debate to reach a final decision. Specifically, we introduce three distinct debating strategies: single debate , team debate , and council debate . In the single debate setting, two LLMs argue from opposing perspectives, while a third model serves as the judge. In team debate, multiple models collaborate on each side; team members first discuss internally before presenting their collective arguments to the opposing team. Finally, in the council debate approach, a group of expert models discuss together to reach a consensus, moderated by a chairperson model. As we observed a strong correlation between the positive classes in Category 2 and Category 3, we adopt a simple heuristic: if our method predicts that a tweet contains a reference to a scientific study or publication (Category 2), we also label it as positive for Category 3 (i.e., containing mentions of scientific entities). In our experiments, we utilize several LLMs, including Gemma3 (12B) Team et al. 2025 , Qwen3 (8B) Yang et al. 2025 , DeepSeek-R1 (8B) Guo et al. 2025 , Phi-4 (14B) Abdin et al. 2024 , Mistral (7B) Jiang et al. 2023 , LLaMA 3.1 (8B) Grattafiori et al. 2024 , o4-mini 1 1 1 https://openai.com/index/introducing-o3-and-o4-mini/ , and Claude-4 2 2 2 https://www.anthropic.com/news/claude-4 . When using only open-source models, the council debate method outperforms other approaches on the development set, and the team debate method is superior to the single debate method. Based on these results, we selected the council debate method as our primary method. In the official rankings, our model ranked 8^{th} ( F_{1}=0.7273 ) in Category 1 (detecting scientific claims) and 9^{th} ( F_{1}=0.7766 ) in Category 3 (detecting mentions of scientific entities). However, it achieved 1^{st} place ( F_{1}=0.7805 ) in Category 2 (detecting references to scientific studies). 2 Related Work 2.1 Scientific Web Discourse Detection Detecting scientific claims and related content in social media, especially on platforms like Twitter 3 3 3 https://x.com , has become a growing area of research in recent years Hafid et al. 2025 ; Alam et al. 2025 ; Hafid et al. 2022 . Shared tasks such as CLEF CheckThat! (Task 4a) Hafid et al. 2025 , SemEval-2023 Task 8 Khetan et al. 2023 , and the FIRE-2023 CLAIMSCAN challenge Sundriyal et al. 2023 have focused on distinguishing scientific claims, references, and science-related entities from general online discourse. Most competitive systems in these tasks use supervised learning, with large pre-trained transformer models like BERT or RoBERTa fine-tuned on carefully annotated datasets, often augmented with auxiliary features such as tweet metadata or ensemble techniques Panchendrarajan and Zubiaga 2024 . While these methods deliver strong results, they rely on large labeled datasets and task-specific training, which can limit adaptability to new domains or languages. In contrast, our approach utilizes the few-shot reasoning capabilities of LLMs and uses a debate-style framework, where multiple models collectively reason and reach a consensus on the presence of scientific discourse, rather than a single model making a prediction. Thus, our methods offer better flexibility and reduce reliance on annotated large training datasets. 2.2 Debating LLMs for Scientific Discourse Detection Recently, debate-based multi-agent frameworks using LLMs have been proposed to improve model reasoning, evaluation, and decision-making Chan et al. 2023 ; Liang et al. 2024 ; Khan et al. 2024 . In these systems, multiple LLMs (sometimes with different roles or perspectives) discuss, argue, or collaborate on a task before producing a final judgment, often resulting in outcomes that align more closely with human assessment than those from a single model. Earlier debate methods typically featured two agents (one ”pro” and one ”con”) with a judge, or small fixed panels, mainly for open-ended generation or evaluation tasks Wang et al. 2023 ; Taubenfeld et al. 2024 ; Estornell and Liu 2024 . However, homogeneous panels can sometimes fall into biased agreement or fail to explore diverse viewpoints Estornell and Liu 2024 . Eo et al. Eo et al. 2025 address efficiency by triggering debates only on uncertain cases. Our council debate method builds on these ideas by using a more diverse set of LLM agents, with each agent contributing its own reasoning. Unlike previous works focused on generation quality evaluation, we apply the debate framework directly to scientific discourse classification in social media. By combining several LLMs in a structured decision process, our approach aims to reduce individual model bias and promote more careful evaluation, resulting in more reliable detection of scientific claims, references, and entities in noisy online environments. 3 Proposed Methods We explored three different multi-LLM approaches for scientific discourse detection, each building upon the previous method’s insights. All approaches use structured deliberation processes but differ in their organizational structure and decision-making mechanisms. In our methods, we use several different prompts. The actual prompts used in our study are given in Appendix A . We apply each method independently for each category. However, when a tweet references a scientific study (Category 2), we also assign a positive label to Category 3 (i.e., containing scientific entities), as such references

Read the complete text