Network-informed Prompt Engineering against Organized Astroturf Campaigns under Extreme Class Imbalance
Nikos Kanakaris, Heng Ping, Xiongye Xiao, Nesreen K. Ahmed, Luca Luceri, Emilio Ferrara, Paul Bogdan · 2025
Read Network-informed Prompt Engineering against Organized Astroturf Campaigns under Extreme Class Imbalance by Nikos Kanakaris, Heng Ping, Xiongye Xiao, Nesreen K. Ahmed, Luca Luceri, Emilio Ferrara, Paul Bogdan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11849.txt.
Introduction
Nikos Kanakaris and six co-authors hunt coordinated astroturf campaigns with language models steered by network-aware prompts, no training required. Their paper from twenty twenty-five tackles a data curse: real campaigns are rare beside normal posts, so classifiers starve on positives. Their frame feeds the model both post text and the social graph around it. A Balanced RAG component fetches examples that offset the imbalance.
Prompt design does the rest of the work. On X data, the setup beats traditional graph-based baselines by two to three times on precision, recall, and F1. The gain comes without fine-tuning a single weight. Disinformation teams gain a detector that adapts by rewording, not retraining, when campaigns mutate.
Catalog record
| CREATOR | Nikos Kanakaris, Heng Ping, Xiongye Xiao, Nesreen K. Ahmed, Luca Luceri, Emilio Ferrara, Paul Bogdan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Network-informed Prompt Engineering against Organized Astroturf Campaigns under Extreme Class Imbalance Nikos Kanakaris, Heng Ping, Xiongye Xiao, Nesreen K. Ahmed, Luca Luceri, Emilio Ferrara, Paul Bogdan Abstract . Detecting organized political campaigns, commonly known as astroturf campaigns, is of paramount importance in fighting against disinformation on social media. Existing approaches for the identification of such organized actions employ techniques mostly from network science, graph machine learning and natural language processing. Their ultimate goal is to analyze the relationships and interactions (e.g. re-posting) among users and the textual similarities of their posts. Despite their effectiveness in recognizing astroturf campaigns, these methods face significant challenges, notably the class imbalance in available training datasets. To mitigate this issue, recent methods usually resort to data augmentation or increasing the number of positive samples, which may not always be feasible or sufficient in real-world settings. Following a different path, in this paper, we propose a novel framework for identifying astroturf campaigns based solely on large language models (LLMs), introducing a Balanced Retrieval-Augmented Generation (Balanced RAG) component. Our approach first gives both textual information concerning the posts (in our case tweets) and the user interactions of the social network as input to a language model. Then, through prompt engineering and the proposed Balanced RAG method, it effectively detects coordinated disinformation campaigns on 𝕏 \mathbb{X} (Twitter). The proposed framework does not require any training or fine-tuning of the language model. Instead, by strategically harnessing the strengths of prompt engineering and Balanced RAG, it facilitates LLMs to overcome the effects of class imbalance and effectively identify coordinated political campaigns. The experimental results demonstrate that by incorporating the proposed prompt engineering and Balanced RAG methods, our framework outperforms the traditional graph-based baselines, achieving 2 × \times -3 × \times improvements in terms of precision, recall and F1 scores. Conference: Companion Proceedings of the ACM Web Conference 2025; April 28-May 2, 2025; Sydney, NSW, Australia Companion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25), April 28-May 2, 2025, Sydney, NSW, Australia DOI: 10.1145/3701716.3717539 ISBN: 979-8-4007-1331-6/2025/04 CCS: Computing methodologies Machine learning algorithms CCS: Computing methodologies Discourse, dialogue and pragmatics Figure 1. Overview of the proposed framework. We first construct a propagation tree G based on the posting and re-posting actions of a social network (e.g. \mathbb{X} /Twitter). Then, we employ graph prompting techniques to encode the produced propagation tree G as a text, in an attempt to make information related to the structure of the social network available to an LLM. A similarity-based balanced RAG component provides several positive (fake news campaigns) and negative examples, which assist in better identifying astroturf campaigns. The text of the original tweet, the text-encoded graph and the examples from the RAG component are then combined with the initial prompt question to form the final prompt. Finally, an LLM predicts whether the given prompt concerns a coordinated disinformation campaign or not. Our framework is flexible and compatible with the most popular prompt engineering and RAG techniques. † † cc-license: by 1. Introduction Social media have undoubtedly emerged as one of the most powerful mediums of information capable of influencing and even shaping public opinion ( Varlamis et al., 2022 ; Fernandez and Alani, 2018 ; Tasnim et al., 2020 ) . They provide their users with an immediate, easy-to-use way of sharing and accessing rich content. Among other factors, their directness and convenience have definitely contributed to their wide adoption. Additionally, they facilitate and promote fruitful conversations among peers, allowing everyone to express their opinion freely ( Grinberg et al., 2019 ; Bhattacharya and Ram, 2012 ) . Despite their widespread success, social media have also provided a fertile ground for malicious users to orchestrate and execute coordinated disinformation campaigns, commonly referred to as astroturf campaigns 1 1 1 In this paper, the phrases ‘astroturf campaigns’ and ‘disinformation campaigns’ are used interchangeably. ( Schoch et al., 2022 ) . Such disinformation campaigns come to augment the negative effects of fake news by spreading persuasive but inaccurate content across a social network. They deliberately confuse and manipulate people and public opinion, usually in an attempt to influence the outcome of a political decision, for instance during a presidential election ( Bovet and Makse, 2019 ; Keller et al., 2017 ; Keller et al., 2020 ) . In most cases, the original coordinators of the campaigns camouflage their true identities by pretending to be trustworthy state representatives, towards promoting their political viewpoints ( Bunker, 2020 ; Freelon and Lokot, 2020 ) . A common characteristic of such campaigns is their close connection to fake accounts. These fake accounts typically include fake, automated, state-sponsored accounts and social bots, which are created to artificially amplify the strength of a piece of information by artificial amplification. Their objective is to distribute content in a way resembling that of organic users ( Keller et al., 2017 ) . Adding to the complexity, it has been observed that specific groups of real users, such as conservative individuals and the elderly, are more susceptible to engaging with and re-sharing political disinformation ( Bovet and Makse, 2019 ; Grinberg et al., 2019 ) . More recently, with the advent of the generative artificial intelligence (GenAI) era, the proliferation and spread of fake news on social media have become even more problematic in modern society ( Ezzeddine et al., 2023 ) . GenAI can further exacerbate the spread of disinformation by equipping malicious individuals with sophisticated tools to create compelling fake content, thus making it increasingly difficult for users to distinguish between real and fake news ( Ezzeddine et al., 2023 ; Loth et al., 2024 ) . In an attempt to mitigate the effects of astroturf campaigns, existing approaches have primarily relied on graph mining and natural language processing (NLP) techniques. Graph neural networks (GNNs) and large language models (LLMs) have emerged as the default technology to perform graph mining and NLP tasks, respectively. The combination of LLMs and GNNs has also been proposed in the literature ( Michail et al., 2022 ; Han et al., 2020 ) . GNNs extract structural and topological features associated with a propagation tree-shaped graph G formed by the (re-)posting actions of the users, while LLMs and other NLP methods deal with the textual content of the posts. The aforementioned propagation tree G usually consists of a set of nodes V denoting the posts and re-posts of the users and a set of edges E denoting the re-posting actions (see Section 3.2 for an extensive problem definition). Most of the existing approaches build on a GNN-based architecture wherein embeddings (vectors) from LLMs are attached to each node of the propagation tree. Briefly, the inclusion of textual embeddings enables GNNs to take into consideration both the content and the user interactions simultaneously. Although recent approaches have generally been successful, there is still space for improvement. In particular, they display several shortcomings: (i) Amounts of labeled data needed: To begin with, they require large amounts of labeled data for training, which may not be available, mainly due to policy restrictions of the considered social media platforms ( Taylor and Pagliari, 2018 ) ; (ii) Class imbalance: Even though the necessary data are available, given that the number of organic