Are Large Language Models Good at Detecting Propaganda?

Julia Jose, Rachel Greenstadt · 2025

Read Are Large Language Models Good at Detecting Propaganda? by Julia Jose, Rachel Greenstadt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13706.txt.

Introduction

Jose and Greenstadt tested whether large language models are good at detecting propaganda in twenty twenty-five. Propaganda uses fallacies and emotional appeals to push an agenda. Spotting its techniques helps readers make informed choices, and automated detection is an active research goal.

The team compared several large models with transformer-based systems on news articles. GPT-4 scored best among the large models but still far below a RoBERTa-based baseline. All three large models beat another baseline at spotting name-calling, one of six techniques tested. Two also beat it on appeals to fear and flag-waving. The authors conclude that large models, despite their fame, lag a smaller specialized model at this task overall.

Catalog record

CREATORJulia Jose, Rachel Greenstadt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13706

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Large Language Models Good at Detecting Propaganda? Julia Jose, Rachel Greenstadt Abstract Propagandists use rhetorical devices that rely on logical fallacies and emotional appeals to advance their agendas. Recognizing these techniques is key to making informed decisions. Recent advances in Natural Language Processing (NLP) have enabled the development of systems capable of detecting manipulative content. In this study, we look at several Large Language Models and their performance in detecting propaganda techniques in news articles. We compare the performance of these LLMs with transformer-based models. We find that, while GPT-4 demonstrates superior F1 scores (F1=0.16) compared to GPT-3.5 and Claude 3 Opus, it does not outperform a RoBERTa-CRF baseline (F1=0.67). Additionally, we find that all three LLMs outperform a Multi-Granularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling), with GPT-3.5 and GPT-4 also outperforming the MGN baseline in detecting instances of appeal to fear and flag-waving. Introduction Propaganda is defined by Jowett & O’Donnell (2006) as “ the deliberate, systematic attempt to shape perceptions, manipulate cognitions, and direct behavior to achieve a response that furthers the desired intent of the propagandist ” ( Jowett and O’Donnell 2018 ) . Manipulative information, as seen in the 2016 US presidential campaign, can harm society by influencing public opinion ( Brattberg and Maurer 2018 ; Golovchenko et al. 2020 ) . Propaganda, however, is not a new concept and dates back to the Second World War. Scholars and educators around then emphasized the need to be able to identify propaganda ( Graham 1939 ; Booth; 1940 ; Hollis; 1939 ) . One of the most prominent detection strategies was to analyze pieces of information for the use of propaganda techniques and to “guard against them” ( Graham 1939 ) . The Institute of Propaganda Analysis (IPA) was formed with this goal in mind. Outlined in their book The Fine Art of Propaganda: A Study of Father Coughlin’s Speeches ( Lee and Lee 1939 ) are seven such propaganda techniques. Examples of some of these techniques include — • Name-Calling: “Giving an idea a bad label and therefore rejecting and condemning it without examining the evidence.” • Bandwagon: “Has as its theme ‘everybody - at least all of us - is doing it!’ and thereby tries to convince the members of a group that their peers are accepting the program and that we should all jump on the bandwagon rather than be left out.” Building on this work, Da San Martino et al. (2019) derived eighteen such propaganda techniques that are widely used in news articles today. Analyzing news articles to check for the presence of propaganda techniques ( Da San Martino et al. 2020 ) could help users gain a deeper understanding of the information’s quality. To automate labeling these techniques in news articles at large scale, we aim to measure the effectiveness of Large Language Models (LLMs) on the task of detecting six out of the eighteen techniques mentioned in Da San Martino et al. (2019). Specifically, we evaluate the following models – OpenAI GPT-3.5, OpenAI GPT-4, and Anthropic Claude 3 Opus under different settings such as zero-shot, one-shot, chain-of-thought prompting ( Wei et al. 2022 ) and other advanced prompting techniques such as generated knowledge promoting ( Liu et al. 2021 ) and self-consistency prompting ( Wang et al. 2022 ) . We find that none of the LLMs outperform a baseline model (a RoBERTa-CRF model with an ensemble of models for final classification layer ( Jurkiewicz et al. 2020 ) both at the macro-F1 level (average across all six techniques) and individual technique F1 scores. GPT-4 outperforms GPT-3.5 and Claude 3 Opus at the macro-F1 level. We also compare the results to another baseline released by the dataset authors ( Da San Martino et al. 2019 ) , which is a Multi-Granularity Network (MGN) model built on BERT embeddings. At the macro-F1 level, we find that GPT-3.5 and Claude 3 Opus do not outperform this MGN model. GPT-4 outperforms this model under one-shot, generated knowledge, and self-consistency prompting mechanisms. Looking at the techniques specifically, we find that all the LLMs outperform the MGN model in detecting instances of name-calling. GPT-4 and GPT-3.5 (only generated knowledge prompting) outperforms this model in detecting appeal-to-fear instances. Similarly, GPT-4 and GPT-3.5 (only one-shot) outperforms this model in detecting instances of flag-waving. None of the LLMs outperform the MGN model in detecting instances of loaded language, doubt, and exaggeration/minimization. Propaganda Device Definition Name-Calling “Labeling the object of the propaganda campaign as either something the target audience fears, hates, finds undesirable or otherwise loves or praises” Loaded Language “Using words or phrases with strong emotional implications to influence an audience” Doubt “‘Questioning the credibility of someone or something” Appeal to Fear “Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments” Flag-Waving “Playing on strong national feeling (or with respect to a group, e.g., race, gender, political preference) to justify or promote an action or idea” Exaggeration or minimization “Either representing something in an excessive manner: making things larger, better, worse (e.g., “the best of the best”, “quality guaranteed”) or making something seem less important or smaller than it actually is ” Note. From ”Fine-Grained Analysis of Propaganda in News Articles”, by Martino et al., 2019, EMNLP-IJCNLP, pp. 5636-5646 ( Da San Martino et al. 2019 ) Table 1: Propaganda Techniques and Definitions used in this study Related Work Language Model Evaluation Language Models have been evaluated on several domain-specific tasks. Wang et al.(2023) evaluated LLMs on natural language understanding in the health domain. They used clinical benchmark datasets and found that GPT-4 outperforms GPT-3.5 and Bard in text classification, named entity recognition, relation extraction, and natural language inference. Rehana et al. (2023) compared LLMs to BERT-based models on protein-protein interaction extraction and found that domain-specific models such as BioBERT and PubMedBERT outperform GPT-4. Li et al. (2023b) similarly evaluated LLMs in the finance domain and found that GPT-4 outperforms all other LLMs in news classification and sentiment analysis. Domain-specific models, however, outperformed GPT-4 in named entity recognition ( Li et al. 2023b ) . Li et al. (2023a) evaluated LLMs in detecting instances of hate speech on social media under different prompt settings and found the results to be better than labels obtained from crowd-sourced workers. Examples where LLMs outperform state-of-the-art models include information extraction ( Wan et al. 2023 ; Ma et al. 2023 ) , question answering ( Bang et al. 2023 ) , machine translation ( Hendy et al. 2023 ; Wang et al. 2023 ) , and information retrieval ( Ziems et al. 2023 ) . Prompt Engineering A prompt is a set of instructions describing the task that you want the LLM to accomplish. Prompts can be engineered to elicit more optimal responses. Zero-shot prompting refers to curating a prompt describing the task that you want the LLM to perform without giving it any examples pertaining to the task. Several studies have looked at using zero-shot prompts with GPT-3.5 and GPT-4 ( Wei et al. 2023 ; Espejel et al. 2023 ) . Few-shot prompting, on the other hand, adds

Read the complete text