UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction
Joe Wilder, Nikhil Kadapala, Benji Xu, Mohammed Alsaadi, Aiden Parsons, Mitchell Rogers, Palash Agrawal, Adam Hassick, Laura Dietz · 2025
Read UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction by Joe Wilder, Nikhil Kadapala, Benji Xu, Mohammed Alsaadi, Aiden Parsons, Mitchell Rogers, Palash Agrawal, Adam Hassick, Laura Dietz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06883.txt.
Introduction
Pulling check-worthy claims out of social media posts is compared across prompting and fine-tuning by Joe Wilder and eight co-authors in a 2025 shared-task paper. The work is the University of New Hampshire entry to the CheckThat! 2025 task for English. Claim extraction is the first step in automated fact-checking pipelines.
Methods tried include few-shot prompting, in-context learning, and fine-tuning across several language model families. The best score on the METEOR metric came from a fine-tuned FLAN-T5 model. Scores did not tell the whole story, though: the authors observe that other methods sometimes produced higher-quality claims despite lower METEOR numbers. That gap between metric and judgment is a useful caution for fact-checking system builders, since a claim can be well worded yet miss what most needs checking.
Catalog record
| CREATOR | Joe Wilder, Nikhil Kadapala, Benji Xu, Mohammed Alsaadi, Aiden Parsons, Mitchell Rogers, Palash Agrawal, Adam Hassick, Laura Dietz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UNH at CheckThat! 2025: Fine-tuning Vs Prompting in Claim Extraction Joe Wilder, Nikhil Kadapala, Benji Xu, Mohammed Alsaadi, Aiden Parsons, Mitchell Rogers, Palash Agrawal, Adam Hassick, Laura Dietz Abstract We participate in CheckThat! Task 2 English and explore various methods of prompting and in-context learning, including few-shot prompting and fine-tuning with different LLM families, with the goal of extracting check-worthy claims from social media passages. Our best METEOR score is achieved by fine-tuning a FLAN-T5 model. However, we observe that higher-quality claims can sometimes be extracted using other methods, even when their METEOR scores are lower. [[email protected],][[email protected],][[email protected],][[email protected],][[email protected],][[email protected], ][[email protected],][[email protected],][[email protected],] † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2025 Working Notes, 9 – 12 September 2025, Madrid, Spain † † address: University of New Hampshire, Durham, NH, 03824, USA 1 Introduction The CheckThat! Lab aims to build computational infrastructure to support human fact-checkers, consisting of a pipeline for verifying the truthfulness of social media text. We participate in Task 2 (English) Sundriyal et al. 2025 , which addresses a specific challenge in this pipeline: Task Statement Given the text of a social media post, extract the main claim in succinct and concise language suitable for a human fact checker to verify. We anticipated a number of unique challenges in this task and designed our methods to study them empirically. Many of our concerns align with Thorne et al. 2018 . Multiple equally relevant claims. Often, multiple extracted claims are topically relevant. However, the task requires focusing on a single fact for manual verification. Analyzing the training and validation data, we found that many gold claims include multiple facts and could be rewritten more succinctly. For example: ”Joe Biden lives in a large estate bought on a senator’s salary.” Missing multi-modal content. We found that many gold claims refer to information not available to participants, such as photos or videos included with the original post but removed during pre-processing. While we accept that claims cannot be based on unseen data, we instruct our models to make educated guesses about the topic. Russia vs Ukraine war https://www.huobi.com/en-us/topic/double-invite/register/? invite_code=ije73223&name=BlackWidow&avatar=6&inviter_id=11343840 BREAKING NEWS LIVE Latest stars: Martin Johnson -200 Latest Supporter: Emeka Efobi Faithful extractions. We are especially concerned about extracted claims where LLM hallucinations introduce content not present in the original post. An example of a hallucinated and overly verbose claim follows: Social Media Post: The salary of a U.S. Senator is $174,000 per year. This is Joe Biden’s house… seems legit :) The salary of a U.S. Senator is $174,000 per year. This is Joe Biden’s house… seems legit :) The salary of a U.S. Senator is $174,000 per year. This is Joe Biden’s house… seems legit :) Hallucinated Extracted Claim Joe Biden’s house, purchased for an amount significantly exceeding the cumulative value of his annual U.S. Senator’s salary of $174,000, raises questions about potential additional, undisclosed sources of income that may have contributed to the down payment, mortgage payments, property taxes, insurance premiums, and ongoing maintenance costs associated with the property. Our emphasis is on exploring the design space across different LLMs and methods—fine-tuning and few-shot prompting—in search of the best trade-off between optimizing the METEOR score and producing claims that are genuinely useful for human fact-checkers. We conduct a broad exploration of methods, prompts, and LLMs, casting a wide net. Our approaches fall into three overarching categories: 1. Fine-tuning approaches, 2. Prompting approaches, and 3. ”Frustratingly easy” baselines. 1 1 1 We did not use baselines provided by the organizer. We describe all explored approaches and submit those performing best on the validation set in terms of METEOR. We only use resources provided by the Task 2 organizers and publicly available large language models from Hugging Face and the Together.AI API service. 2 Approaches: Fine-Tuning and Prompting In this section, we describe methods relying on fine-tuning across LLMs of different parameter scales. Our key takeaway: Flan-T5 Large Chung et al. 2022 offered the best compromise between raw capability and practical fine-tuning feasibility under our hardware and time constraints. 2.1 Finetuned Flan-T5 Large This approach fine-tuned the Flan-T5 Large Chung et al. 2022 model on the CLEF 2025 Task 2 training dataset to align its outputs more closely with the gold-standard claims. Fine-tuning was performed using the huggingface transformers library without advanced techniques such as LoRA or PEFT. Due to resource limitations, billion-parameter models were out of scope. We opted for Flan-T5 Large (783M parameters), which can run locally and is more manageable to train due to its smaller size. A straightforward task-specific prompt was prepended to training examples: Please read the following social media post and extract the claim made within it. Normalize the claim by rephrasing it in a clear and concise manner. Post: $text Extracted Claim: The training ran for 10 epochs on an NVIDIA 4060 GPU, taking nearly four days to complete. This approach’s strength lies in its ability to internalize extraction patterns not easily expressible via prompting alone. It achieved an average validation-set METEOR score of 0.5569. 2.2 2.1 LoRA fine-tuning of Flan-T5 Base The motivation behind this submission was to balance performance with efficiency. Full fine-tuning of T5-Base Chung et al. 2022 gives strong results but incurres high computational costs. Prompt tuning, while efficient, yields limited gains—especially on larger models. LoRA Hu et al. 2021 provides a middle ground by updating only 0.4% of parameters, keeping the rest of the model frozen. LoRA allowed us to adapt the model effectively with minimal overhead. We chose Flan-T5 Base due to its strong baseline performance, aiming to retain quality while reducing resource demands. This model achieved a validation METEOR score of 0.3054 and was our third-best test-set run, with a test METEOR of 0.28. 2.3 Fine-tuned DeepSeek-R1-Distill-Llama-8b Approach Here, we fine-tuned the DeepSeek-R1-Distill-Llama-8b DeepSeek-AI 2025 causal language model on the training dataset to assess its ability to produce reference-style summaries. The training used the following system prompt: Extract the verifiable claim as one sentence from the user input. This approach achieved a validation METEOR score of 0.2541. Fine-tuning yields marginal improvements over a baseline model using a single-shot prompt on the validation set, while zero-shot prompting offers no measurable gain. We conclude that achieving meaningful improvements through fine-tuning—even with moderate-sized 8B-parameter LLMs and using LoRA—would require computational resources beyond our current budget constraints. We compare our fine-tuning methods to prompting-based and in-context learning approaches. These include variations on few-shot prompting, self-refinement Madaan et al. 2023 , self-scoring Bai et al. 2022 , and post-processing. 2.4 Claimifying Social Media Posts with Self-Refinement This method uses a combination of prompting strategies and an iterative Self-Refinement stage. In this stage, the same LLM that generates the initial claim provides feedback based on specific criteria that evaluates the check-worthiness of the claim against the input text. This feedback, along with the input and initial claim, is fed back into the same LLM, which generates a refined version of the claim based on the feedback. We tested both zero-shot and few-shot prompts, with and without a Chain-of-Thought (CoT) trigger phrase. The CoT phrase acts as a cue for the model to reason step