PsychePass : Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments
Zhuang Chen, Dazhen Wan, Zhangkai Zheng, Guanqun Bi, Xiyao Xiao, Binghang Li, Minlie Huang · 2026
Read PsychePass : Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments by Zhuang Chen, Dazhen Wan, Zhangkai Zheng, Guanqun Bi, Xiyao Xiao, Binghang Li, Minlie Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20330.txt.
Introduction
Zhuang Chen and six colleagues present PsychePass, a way to judge how well large language models do therapeutic work. Models show promise in mental health care, but judging their skill is hard because counseling is unstructured and unfolds over long sessions. The authors say current evaluation has an unanchored defect.
That defect shows up as process drift, where a simulated client wanders off the counseling goal, and standard drift, where fixed scoring stays unstable. PsychePass anchors evaluation in trajectories and compares models in tournaments. Steadier measures matter before any model is trusted near real patients.
Catalog record
| CREATOR | Zhuang Chen, Dazhen Wan, Zhangkai Zheng, Guanqun Bi, Xiyao Xiao, Binghang Li, Minlie Huang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PsychePass : Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments Zhuang Chen, Dazhen Wan, Zhangkai Zheng, Guanqun Bi, Xiyao Xiao, Binghang Li, Minlie Huang Abstract While large language models show promise in mental healthcare, evaluating their therapeutic competence remains challenging due to the unstructured and longitudinal nature of counseling. We argue that current evaluation paradigms suffer from an unanchored defect , leading to two forms of instability: process drift , where unsteered client simulation wanders away from specific counseling goals, and standard drift , where static pointwise scoring lacks the stability for reliable judgment. To address this, we introduce PsychePass , a unified framework that calibrates the therapeutic competence of LLMs via trajectory-anchored tournaments. We first anchor the interaction trajectory in simulation, where clients precisely control the fluid consultation process to probe multifaceted capabilities. We then anchor the battle trajectory in judgments through an efficient Swiss-system tournament, utilizing dynamic pairwise battles to yield robust Elo ratings. Beyond ranking, we demonstrate that tournament trajectories can be transformed into credible reward signals, enabling on-policy reinforcement learning to enhance LLMs’ performance. Extensive experiments validate the effectiveness of PsychePass and its strong consistency with human expert judgments. 1 Introduction The explosive growth of large language models offers a beacon of hope for the mental healthcare sector, functioning as scalable, 24/7 counseling assistant support systems, alleviating the heavy burden of professional therapists 23 . However, unlike deterministic tasks such as code generation, counseling is a highly unstructured, longitudinal process inherently reliant on dynamic interaction and systematic stage progression 16 . Consequently, accurately evaluating the therapeutic competence of LLMs remains a pressing challenge in the field 15 . Figure 1: PsychePass anchors two traceable trajectories for calibrating LLMs’ therapeutic competence. Simulating client-therapist interactions is the natural and effective approach for evaluating counseling competence, mirroring the supervision of human counselors. While a few attempts 35 ; 25 adopt this paradigm, we identify a fundamental “ unanchored defect ” that limits their effectiveness. This manifests in two dimensions: 1) Unanchored simulation leads to process drift. Most evaluations rely on single-turn QA or aimless free-form chats based on static client profiles. These unsteered simulations frequently wander without clinical direction, failing to trigger high-stakes scenarios such as trauma processing or skill application. As a result, evaluations capture only superficial features, such as generic empathy, rather than true therapeutic competence. 2) Unanchored judgment leads to standard drift. Existing methods predominantly use static pointwise scoring (e.g., Likert scales), forcing model judges to rely on vague standards without comparative references. This often results in evaluation collapse 27 , where scores cluster in a narrow range with poor discriminability. The resulting lack of clear, comparative signals makes it impossible to accurately rank and optimize models. To overcome the unanchored defect, we introduce PsychePass , a calibration framework for ranking and optimizing the therapeutic competence of LLMs by anchoring two traceable trajectories. 1) Anchoring the interaction trajectory in simulation. To eliminate process drift and ensure the interactions between simulated clients and evaluated models reach sufficient depth, we script interaction trajectories grounded in single-session therapy 24 theory. The simulated client drives the dialogue flow through five complete counseling stages, ranging from alliance building to behavior correction, and performs targeted probing of 12 specific competency dimensions, allowing disentangled measurement of multifaceted capabilities. 2) Anchoring the battle trajectory in judgment. To resolve standard drift and ensure discriminative judgment, we abandon static pointwise scoring. Instead, we construct explicit battle trajectories by determining winners through pairwise battles. We employ a Swiss-system tournament 4 to reduce comparison complexity from N(N-1)/2 to (N/2)\log_{2}N , significantly lowering computational costs. Based on battle results, we utilize the Bradley-Terry model 2 to calculate Elo scores 6 , yielding stable and comparable rankings. Beyond evaluation, we further demonstrate that tournament trajectories can be transformed into credible reward signals to support on-policy reinforcement learning to enhance LLMs’ therapeutic competence, establishing a complete closed calibration loop from evaluation to optimization. For experiments, we first evaluate the therapeutic competence of 12 LLMs, including cutting-edge general models and domain-specific models tailored for counseling, and provide fine-grained, discriminative performance profiles. Extensive analysis validates the effectiveness of PsychePass , achieving high inter-rater agreement with psychologists (Cohen’s \kappa>0.7 ). We then demonstrate that, after being trained with trajectory-based RL, a target LLM achieves a win-to-loss ratio of 62:25 compared to its original counterpart. Our main contributions are: 1) Introducing PsychePass to establish the landscape of therapeutic competence for concurrent LLMs; 2) Exploring how to obtain high-fidelity rewards for aligning LLMs’ counseling capabilities via on-policy RL; 3) Conducting extensive experiments verifying that PsychePass ’s calibration is credible and qualified for serving as a foundation for future work. Systems Simulation Paradigm Judgement Paradigm Optimization Loop Psyche-R1 5 Knowledge QA Accuracy Knowledge RL MentraSuite 28 Knowledge QA Accuracy Knowledge SFT-RL PsychCounsel-Bench 31 Knowledge QA Accuracy Eval Only CBT-Bench 33 Knowledge QA Accuracy Eval Only CounselBench 13 Knowledge QA Pointwise Eval Only ClientCAST 26 Free-form Chat Pointwise Eval Only MindEval 19 Free-form Chat Pointwise Eval Only CARE-Bench 25 Guided Chat Pointwise Eval Only \psi -Arena 35 Guided Chat Pointwise Post-hoc Patch PsychePass (Ours) Scripted Probing Pairwise Trajectory RL Table 1: Comparison of different studies in evaluating therapeutic competencies of LLMs. 2 Related Work Research on LLMs for psychological counseling has expanded rapidly, ranging from synthesizing counseling dialogue corpora 32 ; 29 to incentivizing diagnostic reasoning 10 ; 9 ; 5 ; 28 . However, the real-world counseling process is inherently unstructured and longitudinal, relying on the sophisticated orchestration of diverse therapeutic competencies. Consequently, while the pathway to training conversational agents that can generate superficially plausible counseling dialogues is becoming clear, the rigorous evaluation of their actual clinical effectiveness lags significantly behind their development. Static Knowledge Examination Current assessments predominantly rely on static benchmarks. PsychCounsel-Bench 31 and CBT-Bench 33 utilize professional exams or fixed case studies, while CounselBench 13 focuses on case Q&As. While these benchmarks effectively measure theoretical knowledge, they fall short of verifying whether a model can handle the unstructured, longitudinal dynamics of a real counseling session. Interactive Simulation Tests To move beyond static tests, recent studies develop interactive simulations where LLMs chat with simulated clients 19 ; 26 . However, these environments primarily rely on free-form chats and pointwise scoring, which are prone to process and standard drift. CARE-Bench 25 partially guides client simulation with expert principles but lacks explicit stage-level control. \Psi -Arena 35 simply incorporates counseling stages in prompting yet does not enforce strict process adherence. Moreover, only a few studies incorporate optimization beyond evaluation. Psyche-R1 5 and MentraSuite 28 optimize knowledge reasoning through RL, but their objectives focus on answering exam questions. \Psi -Arena employs post-hoc prompt patching for response revision without internalizing the counseling capabilities. In contrast, PsychePass constrains simulations to explicit probing paths and adopts pairwise comparisons to eliminate process and standard drift, and further establishes a closed loop for evaluation and trajectory-level optimization. We present a straightforward comparison between the proposed PsychePass and existing methods in Table 1 . 3 PsychePass Figure 2: The overall framework of PsychePass . PsychePass calibrates the therapeutic competence of LLMs in three steps (see Figure 2 ): 1) Simulation : Each model interact with scripted clients to generate counseling conversations. 2) Judgment : All