Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning
Michele Joshua Maggini, Dhia Merzougui, Rabiraj Bandyopadhyay, Gaël Dias, Fabrice Maurel, Pablo Gamallo · 2025
Read Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning by Michele Joshua Maggini, Dhia Merzougui, Rabiraj Bandyopadhyay, Gaël Dias, Fabrice Maurel, Pablo Gamallo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07768.txt.
Introduction
Michele Joshua Maggini and 5 co-authors benchmark language models in 2025 on detecting hyperpartisan, fake, polarized, and harmful content. Online platforms struggle with such content, and language models look promising, but no proper benchmark had compared models, usage methods, and languages together. The study spans 10 datasets and 5 languages, from English and Spanish to Arabic and Bulgarian, in binary and multiclass settings.
Strategies range from parameter-efficient fine-tuning to many in-context learning setups: zero-shot prompts, codebooks, few-shot examples chosen randomly or for diversity, and chain-of-thought prompting. The headline discovery is that in-context learning often underperforms compared with fine-tuning a model. That finding, detailed across the full grid of tests, argues that adaptation by training still matters for content moderation, despite the convenience of prompting alone.
Catalog record
| CREATOR | Michele Joshua Maggini, Dhia Merzougui, Rabiraj Bandyopadhyay, Gaël Dias, Fabrice Maurel, Pablo Gamallo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07768
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning Michele Joshua Maggini, Dhia Merzougui, Rabiraj Bandyopadhyay, Gaël Dias, Fabrice Maurel, Pablo Gamallo Abstract The spread of fake news, polarizing, politically biased, and harmful content on online platforms has been a serious concern. With large language models becoming a promising approach, however, no study has properly benchmarked their performance across different models, usage methods, and languages. This study presents a comprehensive overview of different Large Language Models adaptation paradigms for the detection of hyperpartisan and fake news, harmful tweets, and political bias. Our experiments spanned 10 datasets and 5 different languages (English, Spanish, Portuguese, Arabic and Bulgarian), covering both binary and multiclass classification scenarios. We tested different strategies ranging from parameter efficient Fine-Tuning of language models to a variety of different In-Context Learning strategies and prompts. These included zero-shot prompts, codebooks, few-shot (with both randomly-selected and diversely-selected examples using Determinantal Point Processes), and Chain-of-Thought. We discovered that In-Context Learning often underperforms when compared to Fine-Tuning a model. This main finding highlights the importance of Fine-Tuning even smaller models on task-specific settings even when compared to the largest models evaluated in an In-Context Learning setup - in our case LlaMA3.1-8b-Instruct, Mistral-Nemo-Instruct-2407 and Qwen2.5-7B-Instruct. Code and Dataset — https://github.com/HikariLight/hyperpartisanship˙classification/tree/main Introduction Politically biased (PB), hyperpartisan (HP) and fake news (FN) as well as harmful (HF) social media content when covering divisive topics (e.g. politics, COVID-19) present a significant challenge to public discourse and democratic integrity, and most of those phenomena of our interest can fall under the misinformation category ( Wardle and Derakhshan 2017 ) . FN refers to fabricated stories that mimic legitimate news formats ( Lazer et al. 2018 ) . HP, on the other hand, involves misleading coverage of real events presented with a strong partisan bias ( Potthast et al. 2018 ; Maggini et al. 2025 ) . Both often contain politically charged messages that distort facts and polarize audiences. PB reporting further complicates the media landscape by subtly influencing and polarizing public opinion and eroding the impartiality expected of journalism ( Zhou and Zafarani 2020 ) . While the concept of harmful content (HF) is broad and its content can vary based on political priorities ( Eyuboglu et al. 2023 ) , its diffusion on social media can rely on the spread of hate speech ( Yang et al. 2023a ) or misinformation ( Eyuboglu et al. 2023 ) . Indeed, harmful forms like polarizing content are frequently fueled by false or misleading narratives to incite frustration ( Cinelli et al. 2021 ; Osmundsen et al. 2021 ) . Consequently, the detection of such content necessitates robust fact-checking and verification ( Nakov et al. 2022 ) . This conceptual overlap is reflected in recent research challenges. For example, Azizov and Nakov 2023 considered harmful tweet detection a subtask of identifying relevant claims in tweets containing COVID-19 information. By defining a tweet as harmful when it potentially contained misinformation on COVID-19, they effectively demonstrated how, in a specific and critical context, the tasks of detecting harmful content and misinformation are intrinsically linked. Accurate detection of these diverse forms of problematic content is crucial. Large Language Models (LLMs) are valuable tools for this task. The dominant approaches have been fine-tuning (FT) both encoder-only models ( Howard and Ruder 2018 ) and decoder-only LLMs ( Aman 2024 ) . However, a systematic comparison of these models’ performance on the specific tasks of fake and hyperpartisan news, politically biased news, and harmful content detection, especially across multiple languages, is still missing from the literature. Our work fills this gap with a comprehensive comparison of encoder-only and decoder-only LLMs using both fine-tuning (FT) and various in-context learning (ICL) settings. Our ICL methods include: zero-shot prompts with different degrees of task specifications and with rule-based approaches (e.g., codebooks); Few-shot (FS) using both randomly selected examples, and diversity-optimized examples selected through Determinantal Point Process (DPP); Chain-of-Thought (CoT). We conducted experiments on 10 datasets in five different languages, moving beyond the common limitation of using only English or U.S.-centric data ( Maggini et al. 2025 ) . This study addresses three key research questions: • RQ1 : How do different LLM adaptation paradigms (FT vs. ICL) compare for these tasks, considering model architecture, size, and pre-training data? • RQ2 : What is the impact of various ICL strategies—including few-shot examples and rule-based methods—on performance and stability? • RQ3 : How do performance and optimal strategies for these tasks vary across different languages, especially for mid- and low-resource languages? Our extensive experiments reveal that FT remains a highly effective technique, often outperforming ICL strategies. Specifically, fine-tuned decoders performed better for PB and FN detection, while encoders were more effective for HP and HF tweets. Within ICL, the codebook approach was generally the effective, outperforming CoT for classification. The use of DPP for few-shot example selection sometimes reduced classification variance, though it did not consistently boost performance. Related Work Fine-tuning and Political Text Classification Text classification is an NLP task that assigns a label to a given text. FT adapts a pre‑trained model to a specific task by further training the model together with a newly added classification head using task‑specific labeled data. ( Howard and Ruder 2018 ) . While effective, this process can be sensitive to the classifier head’s initialization ( Yang et al. 2022 ) . In political text classification, FT has been successfully applied to various tasks. For instance, Liu et al. 2022 fine-tuned RoBERTa to create POLITICS, achieving state-of-the-art performance on SemEval 2019 for hyperpartisan news detection. Other works have explored stylistic features for hyperpartisan content discrimination ( Potthast et al. 2018 ) , created new datasets for multiclass hyperpartisan detection using fine-tuned BERT models ( Lyu et al. 2023 ) , or combined BERT with ELMo to enhance FT ( Naredla and Adedoyin 2022 ) . More recently, LLMs like Llama 2 have been fine-tuned for tasks such as FN detection, leveraging their understanding and analytical capabilities ( Aman 2024 ; Pavlyshenko 2023 ) . In-Context Learning With the advent of recent decoder-only LLMs, ICL has emerged as a valuable technique in NLP. Users interact with models directly through prompts, specific textual templates containing instructions and optionally examples ( Brown et al. 2020 ) . This approach allows models to perform tasks without prior task-specific fine-tuning ( Efrat and Levy 2020 ) , leveraging a single pre-trained model for various downstream tasks and enabling desired behavior specification via natural language. ICL has shown remarkable performance on challenging reasoning tasks ( Brown et al. 2020 ; Wei et al. 2022 ) . However, ICL is highly sensitive to input format and order ( Lu et al. 2022 ; Min et al. 2022 ) , and can lead to irreproducible outcomes as slight prompt changes significantly impact performance ( Lu, Schuff, and Gurevych 2024 ; Sun, Shaib, and Wallace 2023 ) . To overcome these limitations, our work comprehensively tested various prompt strategies and demonstration selection methods, including DPP for more stable performance, and introduced codebook prompting. Research on prompt design aims to elicit better performance and reasoning. Notable approaches include CoT prompting ( Kojima et al. 2022