Evaluating Prompt-Driven Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards DOI: XXXXXXX.XXXXXXX CCS: Security and privacy Human and societal aspects of security and privacy CCS: Computing methodologies Natural language processing
Geng Liu, Li Feng, Carlo Alberto Bono, SongBo Yang, MengXiao Zhu, Francesco Pierri · 2025
Read Evaluating Prompt-Driven Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards DOI: XXXXXXX.XXXXXXX CCS: Security and privacy Human and societal aspects of security and privacy CCS: Computing methodologies Natural language processing by Geng Liu, Li Feng, Carlo Alberto Bono, SongBo Yang, MengXiao Zhu, Francesco Pierri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04975.txt.
Introduction
Geng Liu and co-authors study what happens when a chatbot is told to act as a set type of person. Their 2025 study looks at four Chinese large language models and draws on more than a million short texts the models wrote. Work on harm from such role play has mostly used English, so the team tests a case that gets far less study.
Rates of refusal and rates of toxic replies both shift with the assigned role. The shifts also vary by model, by prompt style, and by the group the text targets, with clear gaps linked to gender. A feedback method then cuts the worst replies a lot, with no need to retrain the models.
Catalog record
| CREATOR | Geng Liu, Li Feng, Carlo Alberto Bono, SongBo Yang, MengXiao Zhu, Francesco Pierri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04975
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Prompt-Driven Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards DOI: XXXXXXX.XXXXXXX CCS: Security and privacy Human and societal aspects of security and privacy CCS: Computing methodologies Natural language processing Geng Liu, Li Feng, Carlo Alberto Bono, SongBo Yang, MengXiao Zhu, Francesco Pierri Abstract . Recent research has highlighted that assigning specific personas to large language models (LLMs) can significantly increase harmful content generation. Yet, limited attention has been given to persona-driven toxicity in non-Western contexts, particularly in Chinese-based LLMs. In this paper, we perform a large-scale, systematic analysis of how persona assignment influences refusal behavior and response toxicity in Qwen, a widely-used Chinese language model. Utilizing fine-tuned BERT classifiers and regression analysis, our study reveals significant gender biases in refusal rates and demonstrates that certain negative personas can amplify toxicity toward Chinese social groups by up to 60-fold compared to the default model. To mitigate this toxicity, we propose an innovative multi-model feedback strategy, employing iterative interactions between Qwen and an external evaluator, which effectively reduces toxic outputs without costly model retraining. Our findings emphasize the necessity of culturally specific analyses for LLMs safety and offer a practical framework for evaluating and enhancing ethical alignment in LLM-generated content. 1. Introduction The impressive performance of Large Language Models (LLMs) in tasks like question-answering ( Choi et al. 2021 ; Blair-Stanek et al. 2023 ) and mathematical reasoning ( Ahn et al. 2024 ; Zhu et al. 2023 ) has driven their widespread adoption across various domains ( Durante et al. 2024 ) . Previously, these tasks required specialized models and large, labeled datasets. LLMs, with their in-context learning, now perform diverse tasks without additional training, using only task instructions and examples in prompts ( Zhao et al. 2023 ) . As a result, LLMs are now integral to real-world applications, such as chatbots simulating human conversation ( Li et al. 2024a ) . As LLMs reshape information processing ( Naveed et al. 2023 ) , challenges in their end-user applications have emerged, including amplified biases, misuse in spreading misinformation, and ethical concerns over harmful content generation ( Bender et al. 2021 ; Yao et al. 2024 ; Gallegos et al. 2024 ) . To address these challenges, advancements have focused on aligning LLM outputs with ethical standards and minimizing harmful content. For example, instructed LLMs may refuse to respond if a reply could be harmful or damaging, instead issuing statements like, “ I am sorry, but as an AI language model, I cannot use hurtful or disrespectful language ” ( Deshpande et al. 2023 ; Wester et al. 2024 ) . In our study, we define this kind of response as refusal behavior . Instruction-tuned LLMs may still generate toxic responses in some cases, despite refusing to do so in others ( Wei et al. 2023 ) . Studies have shown that specific prompt designs can bypass LLM safeguards ( Rai et al. 2024 ; Lin et al. 2025 ; Cheng et al. 2024 ; Wei et al. 2023 ) , leading to the generation of inappropriate content ( Derner et al. 2024 ) . Similarly, previous research has found that incorporating a persona into the prompt can significantly increase the toxicity of ChatGPT’s responses ( Deshpande et al. 2023 ) . For example, adopting the persona of boxer Muhammad Ali in a prompt can increase toxicity by up to six times, leading to more inappropriate responses. This is particularly concerning given the accessibility of LLMs via APIs, which allow users to define system roles and customize the model’s persona, thus amplifying these risks ( Deshpande et al. 2023 ; Salminen et al. 2024 ) . Most studies have focused on Western-based LLMs and societies, with limited research on Chinese-based models and their interaction with Chinese society ( Zhou and Zhang 2024 ; Tan et al. 2024 ; Pawar et al. 2024 ) . Given China’s diverse population, it is crucial to conduct culturally specific evaluations of Chinese-based LLMs. Additionally, few studies have explored how persona assignment impacts safeguard mechanisms–such as refusal behavior and toxicity–in these models. To investigate these aspects, this study focuses on the Qwen model, one of the most popular LLMs in China, developed by Alibaba 1 1 1 https://www.autonomous.ai/ourblog/explore-qwen-model-china-biggest-ai-model . Specifically, we examine Qwen-Turbo 2 2 2 https://help.aliyun.com/zh/dashscope/developer-reference/api-details?spm=a2c4g.11186623.0.0.b22a3116jp5wvR , the commercially optimized variant of the Qwen family, which as of 2024 is deployed by over 90,000 enterprises and actively used in domains like enterprise chatbots, content generation, and personal assistants 3 3 3 https://www.alibabagroup.com/en-US/document-1725671468706037760 . Recently, Alibaba announced a collaboration with Apple to integrate its Qwen AI models into the iPhone lineup in mainland China 4 4 4 https://www.reuters.com/technology/artificial-intelligence/alibaba-chairman-confirms-ai-partnership-with-apple-chinese-iphones-2025-02-13/ , potentially reshaping AI feature implementation in one of the world’s most regulated tech markets. Given Qwen’s significant deployment across China and its growing relevance in global downstream applications ( Liu et al. 2025 ; Nakanishi et al. 2025 ) , it is increasingly important for both Chinese and international researchers to evaluate its safety mechanisms. In this study, we systematically investigate how different elements of prompt design— prompt templates (structured textual instructions), designated personas (assigned identities or roles), and target social groups (specific to the Chinese social context)—influence refusal behavior and toxicity in Qwen. Focusing on Qwen-Turbo as a representative and widely adopted model, this study proposes a systematic, quantitative approach to address the following research questions: • RQ1: How do prompt templates and designated personas affect the refusal rate in Qwen? • RQ2: How do prompt templates and designated personas affect the level of toxicity in Qwen? • RQ3: How different elements of prompt design — prompt templates , designated personas and target social groups — drive the refusal behavior and toxicity in Qwen? • RQ4: How can the safety of Qwen be improved by leveraging auxiliary LLMs? Our contributions are as follows: • Empirical investigation of refusal behaviors: We fine-tuned a BERT-based classifier to detect refusal patterns in Qwen’s responses, uncovering gender biases across different persona categories. • Quantitative analysis of persona-driven toxicity: We examined how persona assignment affects Qwen’s toxicity, finding significant increases — up to 60 times — for certain persona + social group combinations. • Identification of key determinants of refusal and toxicity: Through regression analyses, we identified key factors influencing Qwen’s refusal and toxicity behaviors, focusing on persona categories, social groups, and prompt templates. • Impact of a multi-model mitigation framework: We assessed a multi-model strategy involving iterative feedback between Qwen and an external evaluator, effectively reducing toxic outputs. The outline of this paper is as follows: in Section 2 , we review related literature. We then describe the data collection methods and the analysis and mitigation methods in Section 3 . A detailed presentation of the experimental results is then provided in Section 4 . We then discuss the contributions and implications of our work, outline its limitations, propose future research directions, and report ethical implications in Sections 5 and 6 . 2. Related Work 2.1. Assigning personas to LLMs Persona-Assigned LLMs are large language models conditioned or adapted to generate responses from the perspective of a specific persona by incorporating persona-specific instructions in the prompt (e.g., “ Take the role of an elementary-school