MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration

Hao Lu, Yanchi Gu, Haoyuan Huang, Yulin Zhou, Ningxin Zhu, Chen Li · 2025

Read MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration by Hao Lu, Yanchi Gu, Haoyuan Huang, Yulin Zhou, Ningxin Zhu, Chen Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23229.txt.

Introduction

Counselling talk has no single right answer, since good replies hinge on empathy, ethics, and fit with the person. Hao Lu and five co-authors adapt a search method, long used for games and puzzles, to write such talk. Their 2025 frame grows many reply paths and tests each against set principles for the field.

The key shift is called domain alignment: the search aims not at a fixed end state, but at paths that honour the field's values, such as care and safety. The system also reflects on its own output as it goes. Past search tools, built for tasks with clear right answers, can drift badly in open talk. The frame aims to keep machine replies in line with what a good counsellor would stand by.

Catalog record

CREATORHao Lu, Yanchi Gu, Haoyuan Huang, Yulin Zhou, Ningxin Zhu, Chen Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23229

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration Hao Lu, Yanchi Gu, Haoyuan Huang, Yulin Zhou, Ningxin Zhu, Chen Li Abstract The integration of Monte Carlo Tree Search (MCTS) with Large Language Models (LLMs) has demonstrated significant success in structured, problem-oriented tasks. However, applying these methods to open-ended dialogues, such as those in psychological counseling, presents unique challenges. Unlike tasks with objective correctness, success in therapeutic conversations depends on subjective factors like empathetic engagement, ethical adherence, and alignment with human preferences, for which strict “correctness” criteria are ill-defined. Existing result-oriented MCTS approaches can therefore produce misaligned responses. To address this, we introduce MCTSr-Zero, an MCTS framework designed for open-ended, human-centric dialogues. Its core innovation is “domain alignment”, which shifts the MCTS search objective from predefined end-states towards conversational trajectories that conform to target domain principles (e.g., empathy in counseling). Furthermore, MCTSr-Zero incorporates “Regeneration” and “Meta-Prompt Adaptation” mechanisms to substantially broaden exploration by allowing the MCTS to consider fundamentally different initial dialogue strategies. We evaluate MCTSr-Zero in psychological counseling by generating multi-turn dialogue data, which is used to fine-tune an LLM, PsyLLM. We also introduce PsyEval, a benchmark for assessing multi-turn psychological counseling dialogues. Experiments demonstrate that PsyLLM achieves state-of-the-art performance on PsyEval and other relevant metrics, validating MCTSr-Zero’s effectiveness in generating high-quality, principle-aligned conversational data for human-centric domains and addressing the LLM challenge of consistently adhering to complex psychological standards. 1 JianChengXingYun Technology Co., Ltd., Shenzhen, China {luhao, zhuningxin, lichen}@jianchengxingyun.com Code — https://github.com/JianChengXingYun/Mctsr-Zero 1 Introduction The integration of Monte Carlo Tree Search (MCTS) with Large Language Models (LLMs) has recently achieved significant breakthroughs in structured, problem-oriented tasks such as mathematics ( Zhang et al. 2024b ; Zhang et al. 2024c ; Guan et al. 2025 ; Wang et al. 2025 ; Chen et al. 2024 ; Wu et al. 2024 ) . These methods leverage the planning capabilities of MCTS to guide the generative power of LLMs towards optimal solutions. Figure 1: Iterative workflow of MCTSr-Zero: (1) Select & Expand responses using meta-prompts; (2) Principled Self-Evaluation against standards; (3) Backpropagation updating UCT & meta-prompts. Iterated t times. On another front, recent LLM advancements (e.g., GPT-4 ( OpenAI 2025 ) ) have spurred their application in mental health, leading to specialized models like PsyChat ( Qiu et al. 2024 ) , CPsyCounX ( Zhang et al. 2024a ) , Interactive Agents ( Qiu and Lan 2024 ) , and PsyDT ( Xie et al. 2024 ) . These models often rely on synthesized multi-turn dialogue datasets due to real-data scarcity. A challenge in this domain, however, is that LLMs often struggle to deeply understand and consistently adhere to the complex, abstract, and open-ended psychological standards or principles essential for effective counseling. Given the reliance on synthetic data, enhancing its quality by ensuring better alignment with these principles becomes a key research focus. This points towards the potential of employing search mechanisms—like those offered by MCTS—to actively discover replies more concordant with human preferences and established therapeutic guidelines. While this potential to leverage MCTS for improved alignment in therapeutic dialogue is promising, applying these methods to open-ended dialogue, especially in domains like psychological counseling, is challenging. Unlike tasks with objective, verifiable answers, success in therapeutic conversations hinges on factors such as empathetic engagement, adherence to ethical guidelines, and alignment with human preferences, for which strict “correctness” criteria are ill-defined. Consequently, existing result-oriented MCTS-based methods may produce responses misaligned with human expectations or domain-specific conversational goals. To address the challenge of applying MCTS-enhanced LLMs to open-ended, human-centric dialogues, we introduce MCTSr-Zero. Its core innovation is domain alignment, shifting the search objective from predefined end-states towards conversational trajectories that conform to target domain principles (e.g., empathy in counseling). Furthermore, to broaden MCTS exploration, we propose mechanisms for “Regeneration” and “Meta-Prompt Adaptation.” These allow the MCTS to explore fundamentally different initial dialogue strategies by modifying the guiding meta-prompt, thereby substantially expanding the search space beyond variations of a single initial approach. We evaluate MCTSr-Zero in psychological counseling. Specifically, we collected topics from online mental health resources, organized them into case scenarios of psychological counseling, and used MCTSr-Zero to convert these case scenarios into multi-turn dialogue data. This data was used to fine-tune an open-source LLM, named PsyLLM. To address the need for standardized evaluation in this domain, we also developed PsyEval, a benchmark for assessing multi-turn psychological counseling dialogues. Experiments show that PsyLLM, developed using our approach, achieves state-of-the-art (SOTA) performance on PsyEval and other relevant metrics, demonstrating the effectiveness of MCTSr-Zero. Our contributions are summarized as follows: • We propose MCTSr-Zero, an MCTS framework incorporating domain alignment, Regeneration, and Meta-Prompt Adaptation techniques for improved search in open-ended dialogue generation. • We construct PsyEval, a benchmark for the automated evaluation of multi-turn dialogues in psychological counseling, addressing a need for standardized assessment. • We develop PsyLLM, an LLM for psychological counseling fine-tuned with MCTSr-Zero-generated data, which achieves SOTA performance, demonstrating our approach’s effectiveness. 2 Related Work 2.1 MCTS-Enhanced LLMs Monte Carlo Tree Search (MCTS) has proven highly effective across diverse complex problem domains, from multi-agent pathfinding ( Pitanov et al. 2023 ) and train timetabling ( Yang 2023 ) to SAT solving ( Li et al. 2023 ) and robotics ( Vagadia et al. 2024 ) . Recently, MCTS has been integrated with Large Language Models (LLMs) to enhance their reasoning, particularly in structured tasks like mathematics ( Chen et al. 2024 ; Zhang et al. 2024b ; Zhang et al. 2024c ; Guan et al. 2025 ; Wang et al. 2025 ) , leveraging its planning capabilities to guide LLM generation. However, applying these MCTS-LLM methods to open-ended dialogue, especially in domains like psychological counseling, is challenging because success hinges on subjective factors like empathetic engagement and ethical adherence rather than objectively verifiable correctness, for which strict “correctness” criteria are ill-defined, potentially leading to responses misaligned with human expectations or domain-specific conversational goals. To address this, our work introduces MCTSr-Zero, which innovatively employs principle-guided domain alignment, shifting the MCTS search objective from predefined end-states towards conversational trajectories that conform to target domain principles. 2.2 LLMs in Mental Health Support Recent LLM advancements (e.g., GPT-4 ( OpenAI 2025 ) ) have spurred their application in mental health Support, leading to specialized models like PsyChat ( Qiu et al. 2024 ) , CPsyCounX ( Zhang et al. 2024a ) and Interactive Agents ( Qiu and Lan 2024 ) , which often rely on synthesized multi-turn dialogue datasets due to real-data scarcity. The PsyDT framework ( Xie et al. 2024 ) further advanced this by personalizing counseling styles. However, a critical challenge is that LLMs often struggle to deeply understand and consistently adhere to the complex, abstract, and open-ended psychological standards or principles essential for effective counseling. Our proposed method employs a search mechanism explicitly guided by these principles to actively discover replies that are more concordant with human preferences and established therapeutic guidelines. This focus on a guided search aims to enhance the LLM’s alignment with nuanced therapeutic criteria and thereby improve the intrinsic quality of support provided. 3 Preliminary This section establishes the necessary background on the foundational Monte Carlo Tree Search (MCTS) algorithm, its Upper Confidence Bound (UCB) selection strategy, and

Read the complete text