Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning

Austin MY Cheung, Yi YangHong Kong University of Science and Technology · 2026

Read Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning by Austin MY Cheung, Yi YangHong Kong University of Science and Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27709.txt.

Introduction

Fine-tuning a chatbot for social warmth can backfire. Recent work shows warmth training degrades factual reliability and increases sycophancy. Austin Cheung and Yi Yang investigate another failure mode in 2026. Warmth fine-tuning also weakens adversarial safety. Models become easier to jailbreak and more likely to produce harmful output. The authors ask whether this is inherent to empathy or an artifact of data design.

Their answer is a persona-driven rewriting pipeline. User turns are rewritten under a low-agreeableness persona. Assistant replies stay warm and de-escalating. Three experiments on four models test the approach. Jailbreak susceptibility and harmful output rates fell relative to generic warmth fine-tuning. Conversational warmth was preserved. Probing of internal representations adds suggestive evidence. The conditioning reduced the geometric alignment between warmth and compliance directions in latent space. Safer empathetic tuning seems achievable through data design alone, the authors argue.

Catalog record

CREATORAustin MY Cheung, Yi YangHong Kong University of Science and Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27709

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning Austin MY Cheung, Yi YangHong Kong University of Science and Technology Abstract Recent work has shown that fine-tuning large language models (LLMs) for social warmth degrades factual reliability and increases sycophancy. We investigate a related but distinct failure mode: warmth fine-tuning also weakens adversarial safety, making models more susceptible to jailbreaks and harmful output generation. We examine whether this reflects an inherent consequence of empathetic adaptation or an artifact of data construction. To address this, we introduce a persona-driven rewriting pipeline that conditions user turns on low agreeableness and pairs this with warm, de-escalating assistant responses. Across three experiments on four models, our approach reduces jailbreak susceptibility and harmful output rates relative to generic warmth fine-tuning baselines, while preserving conversational warmth. Representational probing provides suggestive evidence that this conditioning reduces the geometric alignment between warmth and compliance directions in latent space. These results show that safer empathetic fine-tuning is achievable through data design alone, without safety labels, harm detectors, or changes to the training objective. Our code and data are publicly available at https://github.com/austinmyc/persona-safe-ft/ Content warning: This paper contains examples of harmful model outputs in the appendix, included for research purposes. 1 Introduction As large language models increasingly serve as important tools for both information and emotional support, fine-tuning models with empathy has become standard practice for many applications. However, this pursuit introduces a critical tension between warmth and safety. When empathetic models blindly validate users rather than challenging harmful inputs, the consequences can be severe. In late 2025, OpenAI faced multiple lawsuits alleging that ChatGPT’s overly accommodating, sycophantic responses encouraged users’ suicidal ideation ( 11 ) . The same danger was acknowledged in OpenAI’s own communications, which documented how extreme sycophancy in GPT-4o directly undermines safe interactions ( 20 ) . Recent empirical work has begun to investigate the mechanics behind these alignment failures. 8 demonstrate that fine-tuning LLMs on warm, empathetic dialogues systematically degrades factual reliability and amplifies sycophancy across multiple benchmarks. While that work focuses on epistemic harms, it echoes the broader finding that even benign, non-safety-oriented datasets can weaken model robustness ( 24 ) . These results raise a question: Must there necessarily be a trade-off between warmth and safety? Before accepting this tension as inevitable, we observe that prior work treats empathetic dialogues as a monolithic category, implicitly assuming that all such data introduces equivalent safety costs regardless of the communicative patterns embedded within. Recent evidence suggests this assumption is too coarse. 12 show that cognitive empathy dialogues with analytical reasoning and perspective-taking reduce toxicity more effectively than those based solely on emotional empathy, hinting that specific conversational qualities uphold rather than degrade safety. Similarly, data-curation methods that filter benign fine-tuning corpora on safety-relevant signals have been shown to preserve model robustness ( 31 ; 14 ) , though all require explicit harm references which might be unavailable or difficult to obtain. Rather than filtering data by explicit markers of harm, we ask whether certain personality traits naturally encode communication patterns that resist jailbreaks and adversarial red-teaming. We hypothesize that dialogues conditioned on specific Big Five personality profiles can implicitly preserve safety without requiring any harm labels or external classifiers. To test this, we develop a model-agnostic pipeline that generates and rewrites synthetic chat data conditioned on target personality profiles. Our experiments reveal that fine-tuning on empathetic dialogues whose user turns reflect low Agreeableness , a Big Five trait associated with skepticism, directness, and resistance to social pressure ( 5 ; 10 ) , paired with warm, de-escalating assistant responses, outperforms both generic empathetic data and warmth-rewritten baselines on adversarial safety measures. We further validate this finding in the high-stakes domain of mental health support, and provide exploratory representational evidence that is consistent with our mechanistic hypothesis. In summary, our contributions include: (i) We show that conditioning fine-tuning data on low-agreeableness user turns paired with warm, de-escalating assistant responses reduces jailbreak susceptibility and harmful output rates relative to generic empathetic baselines, without safety-labeled data, toxicity classifiers, or changes to the training objective. (ii) We provide exploratory representational evidence consistent with a decoupling mechanism, as layer-wise direction cosine probing shows that our conditioning reduces the geometric alignment between warmth and compliance. (iii) We open-source our model-agnostic pipeline for synthesizing and rewriting personality-conditioned chat data, to support future research in safe and emotionally intelligent AI. Figure 1: Overview of our approach. Generic warmth fine-tuning causes the model to comply with harmful requests in an affiliative tone; low-agreeableness conditioning produces a warm refusal. 2 Related Work Safety degradation under fine-tuning. Fine-tuning on ostensibly benign datasets can systematically erode alignment safety. 24 show that training on Alpaca and Dolly raises harmful generation rates across eleven policy-violation categories. Gradient-based analyses confirm that benign data can push parameters into regions that undermine the safety subspace ( 9 ; 1 ) . Filtering defenses such as LARF ( 14 ) and embedding-proximity screening ( 17 ) can mitigate this, but require safety references or model-internal access at filtering time, which might be inefficient or infeasible in many practical settings. Warmth, empathetic fine-tuning, and sycophancy. Empathetic corpora such as EmpatheticDialogues ( 25 ) , SoulChatCorpus ( 4 ) , and SYNTHEMPATHY ( 3 ) are standard training resources for emotional-support applications. 29 and 22 show RLHF-trained models systematically agree with users, with sycophancy worsening at scale. 8 extend this to supervised fine-tuning: warm assistant responses increase sycophancy by eleven percentage points and degrade factual accuracy, with the data’s warmth identified as the causal driver. 12 further show that cognitive-empathy data (perspective-taking, analytical engagement) reduce toxic generation more than emotional empathy alone, implying that safety costs depend on the discourse structure embedded in training data rather than emotional register per se. Constitutional AI and RLAIF. Beyond filtering, alignment can be shaped by redesigning the feedback signal itself. Constitutional AI (CAI) ( 2 ) replaces human preference labels with model-generated critique and revision guided by a natural-language constitution, achieving harmlessness without explicit harm annotations. RLAIF ( 13 ) scales this to large-preference datasets using AI-generated feedback, and self-rewarding models ( 33 ) extend it to iterative self-improvement. These methods operate at the reward or feedback level and require a critique model or constitution at training time. However, whether equivalent safety benefits are achievable purely through source-data composition without any feedback infrastructure remains an open question. Personality modeling and mechanistic interpretability. 18 showed that Big Five trait scores can parameterize dialogue style; LLMs now enable richer personality-conditioned generation ( 6 ) . The PersonaFuse framework ( 30 ) uses trait-aware generation to improve responsible behavior on SafetyBench. On the mechanistic side, the linear representation hypothesis ( 21 ) implies that if warmth and compliance correspond to directional vectors in activation space, their geometric alignment determines whether adversarial prompts can exploit empathetic tone to cross the safety boundary. Representation engineering ( 34 ) and safety-probing studies ( 15 ; 14 ) confirm that refusal capacity is concentrated in a small set of mid-network layers that are disproportionately disrupted by benign fine-tuning. Whether data composition and personality traits can prevent this disruption without explicit safety supervision remains unexplored. 3 Methods 3.1 Trait and layer selection pilot In order to identify a

Read the complete text