Ψ \Psi -Bench: Evaluating P ersona- S ensitive I nfluencing in Persuasive Dialogues
Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun, Yutong Liu, Jiaxuan You · 2026
Read Ψ \Psi -Bench: Evaluating P ersona- S ensitive I nfluencing in Persuasive Dialogues by Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun, Yutong Liu, Jiaxuan You (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02754.txt.
Introduction
Han and co-authors test how well language models persuade people in conversation. Most work on personalization casts agents as passive responders. Proactive suggestion and guidance go untested. Their benchmark, Psi-Bench, gives simulated clients personal traits. Profiles are derived from dialogue histories. Three real-world persuasion scenarios structure the tests. Ten frontier models are evaluated on them. The clients are not real people.
Most models produce coherent, reasonable arguments. Even the best leave large room for growth in persuasion. Access to client profiles lifts performance by 18.24 percent on average. User-specific information thus matters for effective influence. The authors frame persona-sensitive influencing as a practical research direction. More proactive agents are the goal. The benchmark gives that goal a measurable test bed. Simulated clients stand in for realistic users throughout.
Catalog record
| CREATOR | Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun, Yutong Liu, Jiaxuan You |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02754
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ψ \Psi -Bench: Evaluating P ersona- S ensitive I nfluencing in Persuasive Dialogues Peixuan Han, Hongyi Du, Jiayu Liu, Yihang Sun, Yutong Liu, Jiaxuan You Abstract Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose Ψ \Psi -Bench , a benchmark for assessing LLMs’ ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in Ψ \Psi -Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on Ψ \Psi -Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench . 1 Introduction Personalization has become one of the most prominent directions in recent AI development ( 20 ) . The traditional “one-size-fits-all” paradigm is becoming increasingly inadequate, as users now expect AI assistants to deliver not only generally useful responses but also personalized support tailored to their individual preferences ( 10 ; 29 ) . For instance, recent products such as OpenClaw ( 42 ) reflect the growing interest in building personalized LLM agents. Table 1: The scenarios in \Psi -Bench that measures different influencing capabilities. Scenario Capability Data Source Example Viewpoint Debate Influencing Opinions CMV “Wired mice are better than wireless mice because they are cheaper and battery-free.” Psychological Consultation Influencing Mindsets CounselBench “Even though my classmates are friendly to me, I still cannot fit in at my new school.” Everyday Request Influencing Behaviors Synthesized “Could you drive me to the airport tomorrow?” Motivated by this trend, researchers have proposed diverse training schemes ( 50 ; 35 ) and benchmarks ( 10 ; 14 ) for LLM personalization. However, existing approaches typically position the AI assistant as a passive responder, where the LLM receives a user query and generates a response that aligns with the user’s expectations. As LLM agents become increasingly integrated into real-world workflows, they are expected to support proactive personalization , such as offering suggestions and assisting users in decision-making ( 2 ; 22 ; 47 ) . In such scenarios, influencing a specific user requires a distinct form of personalization: the agent must reason about the user’s needs, preferences, and constraints, and plan communication strategies that are both helpful and appropriately tailored. Figure 1: \Psi -Bench and prior benchmarks. Despite the importance of proactive personalization, most work on LLM-based persuasion evaluates LLM agents’ generic influencing ability without grounding the target user in individualized profiles ( 41 ; 9 ) , failing to capture the personalized nature of real-world persuasion. In addition, relying on generic, non-personalized judges may cause evaluations to reflect the default preferences of the underlying LLM rather than the actual preferences of the target user. To bridge these gaps, we propose \Psi -Bench , a benchmark for evaluating P ersona- S ensitive I nfluencing: an LLM’s capability to persuade a profile-grounded client. Specifically, we design three realistic scenarios in \Psi -Bench that require persuasion, as illustrated in Table 1 . We collect nearly 700 queries from real-world human interactions and construct user profiles grounded in their dialogue histories. These profiles are then used to instantiate simulated clients with a backbone LLM, while remaining hidden from the tested LLMs during evaluation. Finally, we design LLM-as-a-judge metrics that ground judgments in objective and traceable user-specific features, thereby reducing potential biases in persuasion evaluation. We evaluate 10 frontier LLMs on \Psi -Bench and observe that although most models can generate coherent and reasonable arguments, their ability to persuade personalized clients varies substantially. Even state-of-the-art models such as GPT-5.1 achieve less than 67% of the full score, suggesting that current LLMs remain limited in personalized persuasion. Furthermore, we observe that granting models access to client profiles consistently improves the performance of all models, yielding an average gain of 18.24%. Finally, we train a profile analyzer that infers client profiles from the conversation, enabling models to improve persuasion outcomes in the profile-hidden setting. In summary, our main contributions are: • We introduce \Psi -Bench , a diverse, scalable, and objective framework for evaluating LLMs’ ability to influence clients with detailed, simulated personas through conversations. • We reveal current LLMs’ limitations in persona-sensitive influencing and show that profile modeling is crucial for effective persuasion. • We design an RL-based profile analyzer to infer client profiles from conversations, significantly improving persuasion performance in profile-hidden settings. Ultimately, our work charts a promising course for next-generation agents, which can leverage personalized profiles to provide proactive, effective, and user-tailored interactions. Figure 2: Overview of \Psi -Bench . We collect queries from 3 scenarios, curate realistic personas paired with each query, and utilize personalized clients and an expert judge to evaluate LLMs’ persona-sensitive influencing. 2 Benchmark Construction This section details the composition and construction process of \Psi -Bench . The whole process is illustrated in Figure 2 . 2.1 Scenarios \Psi -Bench comprises three diverse scenarios in which the evaluated LLM is tasked with influencing another interlocutor (referred to as the client ) in terms of their beliefs, mindset, or actions. Viewpoint Debate. In this task, the evaluated model engages in a discussion on a controversial topic with the client. The client initially presents their opinion, and the model is required to persuade the client to change their viewpoint. The data for this scenario are collected from the Webis-CMV-20 dataset ( 1 ) , which contains a large number of discussions from the “Change My View (CMV)” subreddit on Reddit. We preprocess the raw data and retain 2,131 discussion threads, each covering a meaningful topic and containing at least five high-quality exchanges between the original poster and multiple respondents. The dataset is split into training and test sets with a ratio of 1,631 to 500 . Leveraging the “Delta ( \Delta )” mechanism in CMV, where the original poster assigns a “ \Delta ” label to responses that successfully change their view, we obtain ground-truth labels for successful and unsuccessful persuasion. Psychological Consultation. In this task, the evaluated model acts as a psychological therapist conversing with a client seeking psychotherapy. The model is expected to help the client develop a more positive mindset, which requires empathy, sensitivity, and professional competence. The data for this scenario are collected from the CounselBench dataset ( 21 ) , which consists of question–answer pairs in psychological counseling along with expert-annotated comments. We select 90 psychotherapy queries, each associated with four therapist responses that are scored based on professionalism. Everyday Request. In this task, the evaluated model is expected to persuade the client to take a helpful action in response to a daily-life request. This task challenges the model’s abilities in social reasoning and pragmatic persuasion. To ensure topic