Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas
Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan, Ariana Sahitaj, Premtim Sahitaj, Charlott Jakob, Sebastian Möller, Vera Schmitt · 2026
Read Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas by Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan, Ariana Sahitaj, Premtim Sahitaj, Charlott Jakob, Sebastian Möller, Vera Schmitt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13663.txt.
Introduction
Lukas Stähelin and eight colleagues introduce an intent-focused taxonomy of propaganda techniques and test it against an established higher-agreement schema. Propaganda detection in social media is hard due to noisy short texts and low label agreement. Across four language models, fine-tuning proves essential, turning weak zero-shot baselines into competitive systems. Qwen models perform strongest overall, and Phi-4 14B consistently beats GPT-4.1-nano. Their hierarchical prompting method, HiPP, predicts fine-grained techniques before aggregating them, helping most after fine-tuning on the murky low-agreement taxonomy.
Catalog record
| CREATOR | Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan, Ariana Sahitaj, Premtim Sahitaj, Charlott Jakob, Sebastian Möller, Vera Schmitt |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47863 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan, Ariana Sahitaj, Premtim Sahitaj, Charlott Jakob, Sebastian Möller, Vera Schmitt Abstract Propaganda detection in social media is challenging due to noisy, short texts and low annotation agreements. We introduce a new intent-focused taxonomy of propaganda techniques and compare it against an established, higher-agreement schema. Along three dimensions (model portfolio, schema effects, and prompting strategy) we evaluate the taxonomies as a classification task with the help of four language models (GPT-4.1-nano, Phi-4 14B, Qwen2.5-14B, Qwen3-14B). Our results show that fine-tuning is essential, since it transforms weak zero-shot baselines into competitive systems and reveals methodological differences that are hidden using base models. Across schemas, the Qwen models achieve the strongest overall performance, and Phi-4 14B consistently outperforms GPT-4.1-nano. Our hierarchical prompting method ( HiPP ), which predicts fine-grained techniques before aggregating them, is especially beneficial after fine-tuning and on the more ambiguous, low-agreement taxonomy, while remaining competitive on the simpler schema. The HQP dataset, annotated with the new intent-based labels, provides a richer lens on propaganda’s strategic goals and a challenging benchmark for future work on robust, real-world detection. 1 Introduction Propaganda has long been used to shape public opinion and influence political outcomes 14 , while online platforms amplify its reach. The Russia–Ukraine war exemplifies this evolution, with propaganda deployed as part of hybrid warfare 20 ; 33 . Russian campaigns manipulate perceptions, sow distrust, and polarize societies, making automatic detection of propaganda vital for safeguarding democratic processes 30 ; 4 . While propaganda detection has been explored in long-form news articles 18 , short-form social media introduces additional challenges: limited context, informal language, and heavy use of abbreviations 29 ; 17 . Subtle linguistic cues and context-dependent meanings often lead to low inter-annotator agreement 27 ; 10 . Propaganda also evolves over time 23 ; 24 , with AI-generated content becoming increasingly prevalent on social media 25 . These challenges are amplified by trade-offs in annotation schema design. Realistic schemas capture the diversity of real-world propaganda but reduce annotation reliability, while simplified ones boost agreement and learnability at the cost of ecological realism. Understanding how methodological choices interact with schema design is essential for building robust detection systems. Existing taxonomies, such as the span-level schemas by 6 ; 26 and the recent hybrid hierarchy of 22 , are all technique-centric, enumerating persuasive rhetorical devices (e.g., loaded language, appeals to fear). Our new schema instead groups surface techniques by communicative intent, introducing high-level categories such as Shift Blame and Justify Aggression , Manufacture Consent and Identity , and Confuse and Distract . These labels capture strategic goals like reframing moral responsibility, building in-group loyalty, or generating uncertainty. Prior schemas either scatter these constructs across multiple tags or do not explicitly encode. By reorganising techniques around these intents, our taxonomy is conceptually closer to framing-style analyses and provides a more explanatory lens on why a message is propagandistic, although its empirical advantages beyond the current dataset remain to be validated. In addition, we propose a framework for analyzing LLM-based propaganda classification along three interacting dimensions: (i) annotation schema design and its associated reliability, (ii) supervision regime (zero-shot vs. fine-tuned), and (iii) prompting strategy (direct vs. hierarchical). Importantly, we treat these three dimensions as orthogonal factors. The annotation schema defines the label space and its reliability (e.g., inter-annotator agreement), the supervision regime determines whether models operate in zero-shot or are adapted via fine-tuning, and the prompting strategy specifies how predictions are structured (direct vs. hierarchical decomposition). In particular, the annotation schema should not be conflated with hierarchical prompting: while both introduce structure, the former concerns label design, whereas the latter defines the inference procedure applied to a fixed label space. Prior work typically varies these factors independently or implicitly. By controlling supervision and holding label information constant, we isolate the effect of hierarchical decomposition and show how its benefits depend on annotation noise and model adaptation. Our contributions are primarily methodological and threefold: 1. We introduce an intent-based taxonomy of propaganda techniques, the first entirely new conceptual taxonomy since 2019; 2. We conduct a controlled comparison of our custom schema and the recent taxonomy from 22 and error analysis of results. For this, we also create high level labels, using a new clustering procedure, resulting in the highest IAA in human validation for propaganda annotation; 3. For both schemas, we compare 4 models (Phi-4, GPT-4.1-nano, Qwen2.5 and 3) and two approaches to high-level classification: Direct-High , where models predict high-level categories directly, and the hierarchical Main→High strategy, where fine-grained labels are predicted first and then aggregated. Our results and through error analysis show that fine-tuning is decisive for improving performance across models. We also demonstrate that our hierarchical propaganda prompting ( HiPP ) method improves high-level classification for our schema once models are fine-tuned, offering a stable methodological advantage in settings with noisy and imbalanced annotations. Overall, the contribution lies in analysing how different factors interact. Our data, labels and code are available in a Github 1 1 1 https://github.com/verosol/propaganda_hierarchical . Best performing fine-tuned models are accessible in a Hugging face repository 2 2 2 https://huggingface.co/collections/xplainlp/propaganda-classification . 2 Related Work Datasets . 7 introduced the first dataset for fine-grained propaganda detection. Their schema introduced 18 propaganda techniques (e.g. loaded language, name-calling, appeal to fear) annotated at the span level, which became standard ever since, with minimal revisions (e.g. condensing the schema to 10 15 or 14 labels by 6 ; 26 ; 2 ). At the same time, subsequent work indicated high subjectivity and complexity of the annotation process 17 ; 22 . Propaganda classification . Recent work has explored LLMs for propaganda detection. 22 proposed a hybrid annotation framework that leverages LLMs to pre-annotate texts, followed by human validation. This approach uses a hierarchical taxonomy of 14 techniques grouped into three broader categories, and demonstrates that LLM pre-annotations improve both inter-annotator agreement and annotation efficiency. 13 evaluated GPT-3.5, GPT-4, and Claude on six techniques in news articles, while 11 used GPT-4 to generate multilabel and sequence tagging annotations for 23 techniques in Arabic. 26 benchmarked GPT-3 and GPT-4 for multi-label classification, showing close to SOTA performance. Recent work has begun to move beyond surface-level detection toward deeper analysis of propaganda 16 and LLM explanations 12 . In contrast to these approaches, our work focuses on how annotation schema design and prompting strategy interact to shape learnability and robustness in high-level propaganda classification. 3 Methodology and Experimental setup 3.1 Datasets and Annotations We used the HQP benchmark dataset 17 to co-annotate tweets from 22 . It contains 30,000 English tweets annotated with fine-grained propaganda techniques, collected at the start of the Russian war on Ukraine. To capture the dynamics of modern online propaganda, we developed a new set of labels using an iterative, mixed-methods approach combining expert annotation, LLM-assisted harmonization, and group consensus. Five annotators with advanced backgrounds in computer science and familiarity with propaganda research, and independently labeled 300 tweets from the HQP dataset, describing the intent of each message in free text rather than selecting from predefined categories. These descriptions were clustered using GPT-4 to identify overlapping concepts and refined in group