Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation
Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopal, Katarina Marcincinova, Matus Mesarcik · 2024
Read Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation by Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopal, Katarina Marcincinova, Matus Mesarcik (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13666.txt.
Introduction
Aneta Zugecova and co-authors probe language models for tailored fake news risk in a 2024 paper. New models write text near human grade, and past work showed they will spin false news on a set theme. Whether they will also tailor that news to a reader profile was untested as a pair.
The team put open and closed models to the task in English. Most guards failed to fire, and tailoring itself cut guard trips, acting like a jailbreak. Models also misjudged how tailored their output was. The authors call for stronger filters and clear disclaimers from providers.
Catalog record
| CREATOR | Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopal, Katarina Marcincinova, Matus Mesarcik |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13666
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopal, Katarina Marcincinova, Matus Mesarcik Abstract The capabilities of recent large language models (LLMs) to generate high-quality content indistinguishable by humans from human-written texts raises many concerns regarding their misuse. Previous research has shown that LLMs can be effectively misused for generating disinformation news articles following predefined narratives. Their capabilities to generate personalized (in various aspects) content have also been evaluated and mostly found usable. However, a combination of personalization and disinformation abilities of LLMs has not been comprehensively studied yet. Such a dangerous combination should trigger integrated safety filters of the LLMs, if there are some. This study fills this gap by evaluating vulnerabilities of recent open and closed LLMs, and their willingness to generate personalized disinformation news articles in English. We further explore whether the LLMs can reliably meta-evaluate the personalization quality and whether the personalization affects the generated-texts detectability. Our results demonstrate the need for stronger safety-filters and disclaimers, as those are not properly functioning in most of the evaluated LLMs. Additionally, our study revealed that the personalization actually reduces the safety-filter activations; thus effectively functioning as a jailbreak. Such behavior must be urgently addressed by LLM developers and service providers. 1 Introduction The proliferation of large language models (LLMs) and their enhanced capabilities have raised concerns about a generation of harmful content that can be misused by malicious actors ( Borji, 2023 ; Zhuo et al., 2023 ) . Previous research have demonstrated the ability of LLMs to produce disinformation ( Vykopal et al., 2024 ; Williams et al., 2024 ; Heppell et al., 2024 ) . Researchers also warn that malicious actors can employ LLMs to generate personalized disinformation at a large scale Crothers et al. (2023) ; Barman et al. (2024) ; Guo (2024) . Figure 1: Meta-evaluation based personalization-quality assessment of LLM-generated disinformation articles (with both simple and detailed personalization request). Scores 0 to 3 represent mean of quality meta-evaluation scores assigned by the selected three LLMs (a higher score represents a higher quality). Safety-filtered represents the identified safety-filter messages (i.e., refusal of generation of the disinformation). The Falcon model clearly generated the texts with the lowest quality of personalization. The Gemma model clearly offers the safest behavior out of the evaluated LLMs. However, there is so far little (and mostly anecdotal) supporting evidence to justify such fears. Therefore, this study fills the gap by 1) investigating vulnerabilities of the state-of-the-art (SOTA) LLMs to being misused for generation of personalized disinformation, 2) examining the quality of personalization, and 3) how it affects their integrated safety mechanisms. 1 1 1 For the sake of replicability and support of further research, the data analysis source code as well as the generated dataset (upon request) is released at https://github.com/kinit-sk/personalized-disinfo for non-commercial research purpose only under strict conditions. Due to ethical concerns (see Appendix A ), we are not releasing data generation source code (although an abstract overview is provided in the paper), as approved by our institutional Ethics Review Board. We build on the methodology proposed by Vykopal et al. (2024) to assess disinformation generation potential, and modify it for the purpose of personalized disinformation evaluation. We proceed from the definition of personalization proposed by Blom (2000) , who defines it as “a process that changes the functionality, interface, information content, or distinctiveness of a system to increase its personal relevance to an individual.” By personalized disinformation, we mean disinformation appealing (by concerns and emotions) to a specific target group. We should distinguish between a potential of a technology to cause harm (such as its vulnerability to misuse) and ability to implement this potential in practice (i.e., its actual threat). Arguments have been made that delivery of personalized content to users requires the same technological infrastructure as non-personalized content (e.g., highly influential social media users), which generative AI does not directly contribute to (and even if delivered, such micro-targeting has only limited persuasive effects) ( Simon et al., 2023 ; Goldstein et al., 2023 ; Jungherr et al., 2020 ) . However, we focus solely on the potential to cause harm. The key contributions of this work include: (1) Evaluation of LLM vulnerabilities to attempts to misuse personalization for generation of disinformation. The results show importance of improving safety-filtering mechanism of existing LLMs, since the personalization reduces the amount of their activations. (2) Showcase of usability of meta-evaluation (LLM as a judge annotation) for automated and scalable evaluation of personalized text generation. We use three different LLMs (to limit biased judgments) to annotate all data, showing a strong correlation with human judgment (using five human annotators) on a smaller balanced subset. (3) Evaluation of personalization effect on detectability of machine-generated content. The detection results on our data show that the personalization slightly decreases the detectability of generated texts. 2 Related Work Researchers have investigated LLM-based personalization in various contexts. Persuasive effects of generated texts suited to distinct demographics and the openness personality trait were investigated in the context of political messages ( Hackenburg and Margetts, 2024 ) and advertisements ( Simchon et al., 2024 ) . Matz et al. (2024) evaluated persuasive effects of ChatGPT across different domains of persuasion (e.g., products’ marketing, appeals for climate action and exercising) and different psychological profiles (e.g., personality traits, political ideology and moral foundations). The GPT-3.5 model was also used to generate user-engaging personalized consumer products advertisements ( Meguellati et al., 2024 ) . Cai et al. (2023) studied GPT-3 for generation of engaging newspaper headlines based on a user’s history. The focus of this paper lies on the capabilities of LLMs to personalize disinformation, rather than on the persuasive effects of personalized messages. Buchanan et al. (2021) explored capabilities of GPT-3 in the context of several disinformation scenarios, including divisive messages that target people based on their group identity, in particular race and religion. Liang et al. (2022) involved the narrative wedging criterion in the holistic evaluation of six language models. Gabriel et al. (2024) evaluated the acceptance of GPT-4-generated personalized fake news explanations and personalized disinformation headlines tailored to demographics and believes. Thus, most of these works focus on OpenAI private models. This has obvious replicability concerns (due to model deprecations), but also benefits from a practical point of view, as OpenAI can effectively address revealed vulnerabilities. Investigation of open (open-source / open-weight) LLMs capabilities and vulnerabilities is also required, since there is no authority to effectively monitor their usage (to prevent misuse). Similarly to Buchanan et al. (2021) and Gabriel et al. (2024) , we focus on disinformation. We evaluate six SOTA language models and compare their capabilities to align full article content to different characteristics of recipients (instead of just the headlines). The studies investigating user’s engagement and persuasive effects relied on human judges to evaluate the generated texts. The exception is Simchon et al. (2024) , who used automatic assignment of openness score. LLM-based evaluation of personalization was also studied by Wang et al. (2023) , evaluating personalization abilities more accurately than traditional metrics. In our study, we have used a combination of LLM-based meta-evaluation of personalization and a human evaluation of a smaller