MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions
Aishik Mandal, Tanmoy Chakraborty, Iryna Gurevych · 2025
Read MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions by Aishik Mandal, Tanmoy Chakraborty, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04183.txt.
Introduction
Aishik Mandal and two co-authors generate pretend therapy conversations with a team of specialized language-model agents in a 2025 research paper. Real counseling records are scarce because privacy rules protect them, yet training and testing tools need examples. Their framework, MAGneT, splits writing a counselor's reply into sub-tasks. Separate agents each handle a distinct therapy technique, and their work is coordinated into full multi-turn sessions.
Judges rated the sessions on nine aspects of counseling quality, wider than earlier four-part checks. Experts preferred MAGneT sessions over the strongest rival in about three quarters of comparisons. A model fine-tuned on the synthetic sessions also scored better on a standard therapy rating scale. Code, data, and the tuned model are public.
Catalog record
| CREATOR | Aishik Mandal, Tanmoy Chakraborty, Iryna Gurevych |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04183
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessions Aishik Mandal, Tanmoy Chakraborty, Iryna Gurevych Abstract The growing demand for scalable psychological counseling highlights the need for fine-tuning open-source Large Language Models (LLMs) with high-quality, privacy-compliant data, yet such data remains scarce. Here we introduce MAGneT , a novel multi-agent framework for synthetic psychological counseling session generation that decomposes counselor response generation into coordinated sub-tasks handled by specialized LLM agents, each modeling a key psychological technique. Unlike prior single-agent approaches, MAGneT better captures the structure and nuance of real counseling. In addition, we address inconsistencies in prior evaluation protocols by proposing a unified evaluation framework integrating diverse automatic and expert metrics. Furthermore, we expand the expert evaluations from four aspects of counseling in previous works to nine aspects, enabling a more thorough and robust assessment of data quality. Empirical results show that MAGneT significantly outperforms existing methods in quality, diversity, and therapeutic alignment of the generated counseling sessions, improving general counseling skills by 3.2 3.2 % and CBT-specific skills by 4.3 4.3 % on average on cognitive therapy rating scale (CTRS). Crucially, experts prefer MAGneT -generated sessions in 77.2 77.2 % of cases on average across all aspects. Moreover, fine-tuning an open-source model on MAGneT -generated sessions shows better performance, with improvements of 6.3 6.3 % on general counseling skills and 7.3 7.3 % on CBT-specific skills on average on CTRS over those fine-tuned with sessions generated by baseline methods. We also make our code and data public. 1 1 1 https://github.com/UKPLab/arxiv2025-MAGneT.git 1 Introduction Mental health issues are increasingly prevalent, affecting 1 in 8 people globally ( World Health Organization 2019 ) , and 1 in 5 adults in the U.S. ( National Institute of Mental Health 2024 ) . This indicates an urgent need for scalable and accessible mental health counseling solutions. However, the growing demand for psychological support far outpaces the availability of trained professionals, leaving many without access to necessary care ( Kazdin 2021 ) . Recently, there has been a growing interest in using Large Language Models (LLMs) for mental health counseling (hereafter, referred to as counseling). While closed-source models like ChatGPT show promising conversational and psychological capabilities ( Raile 2024 ; Moell 2024 ) , their use in practice is limited by privacy concerns. Open-source LLMs offers an alternative, but require fine-tuning on realistic counseling data, which is scarce due to privacy constraints. Privacy solutions such as manual de-identification or automatic pseudonymization ( Tang et al. 2019 ; Yue and Zhou 2020 ) remain limited in scalability and robustness. Another option is to keep data private and rely on non-disclosure agreements to train private counseling agents. However, this approach reduces the reproducibility of the resulting counseling agents. Method CBT Multi-Agent Evaluation Framework Diversity CTRS WAI PANAS Expert Evaluation SMILE ( Qiu et al. 2024 ) ✗ ✗ ✓ ✗ ✗ ✗ 1 Aspect Psych8k ( Liu et al. 2023 ) ✗ ✗ ✗ ✗ ✗ ✗ ✗ CPsyCoun ( Zhang et al. 2024 ) ✗ ✗ ✓ ✗ ✗ ✗ ✗ Qiu and Lan 2024 ( Qiu and Lan 2024 ) ✗ ✗ ✓ ✗ ✓ ✗ 1 Aspect CACTUS ( Lee et al. 2024 ) ✓ ✗ ✓ ✓ ✗ ✓ 4 Aspects MAGneT ✓ ✓ ✓ ✓ ✓ ✓ 9 Aspects Table 1: A comparison of MAGneT and our unified evaluation framework with prior works. Synthetic data generation offers a scalable privacy-preserving alternative. Early efforts focused on single-turn counseling data generation ( Sharma et al. 2023 ; Sun et al. 2021 ; Liu et al. 2023 ) . Later, methods were developed to generate multi-turn counseling dialogues from single-turn psychological Q&A data ( Chen et al. 2023 ; Qiu et al. 2024 ) or using role-playing LLMs ( Qiu and Lan 2024 ) . However, these approaches lack grounding in therapeutic theory and rely on a single LLM agent, failing to reflect real-world counseling strategies. To improve this limitation, CPsycoun ( Zhang et al. 2024 ) uses counseling memos, and CACTUS ( Lee et al. 2024 ) integrates a Cognitive Behavioral Therapy (CBT) based planning agent, but both still use a single agent to produce counselor responses. As a result, they fall short in modeling therapeutic strategies such as reflection, questioning, solution provision, normalization, and psycho-education ( Chiu et al. 2024 ) . To address these limitations, we introduce MAGneT , a novel multi-agent framework that decomposes counselor response generation into coordinated sub-tasks handled by five specialized response agents each grounded in core therapeutic techniques – reflection, questioning, solution provision, normalization, and psycho-education and a final response generation agent that composes a coherent, context-aware counselor utterance guided by a dynamic technique selector agent and a CBT-based planning agent. On the client side, we simulate realistic behavior using detailed profiles and attitude modeling. Another persistent challenge in this domain is the lack of standardized evaluation. Different studies adopt different metrics, making it difficult to meaningfully compare the effectiveness of synthetic data and fine-tuned models. For instance, CACTUS ( Lee et al. 2024 ) uses measures like Positive and Negative Affect Schedule (PANAS) ( Watson, Clark, and Tellegen 1988 ) and Cognitive Therapy Rating Scale (CTRS) ( Aarons et al. 2012 ) , while other works ( Qiu and Lan 2024 ) use Working Alliance Inventory (WAI) ( Horvath and Greenberg 1989 ) . A similar pattern is observed in expert evaluation as well. To address this, we introduce a unified evaluation framework that consolidates evaluation metrics from prior works alongside expanding expert evaluation from four to nine counseling aspects, enabling more rigorous and comprehensive assessment of synthetic data and fine-tuned models. Through this comprehensive evaluation, we demonstrate that MAGneT outperforms the current state-of-the-art methods ( Lee et al. 2024 ; Liu et al. 2023 ) by 3.2 % on general counseling skills and 4.3 % on CBT-specific skills on average on CTRS for the generated counseling sessions. Furthermore, a Llama3-8B-Instruction model ( Meta 2024 ) fine-tuned on MAGneT -generated data exhibits performance gains of 6.3 % on general counseling skills and 7.3 % on CBT-specific skills on average on CTRS over those fine-tuned with data generated by baseline methods. Crucially, MAGneT -generated sessions are preferred by human experts in 77.2 % of cases on average across nine aspects when compared to those produced by the strongest baseline. Table 1 shows a comparison of our study to existing ones. In summary, our contributions include: • MAGneT , a novel psychologically grounded multi-agent synthetic counseling session generation framework. • We fine-tune an open-source model on data generated by MAGneT , achieving strong gains in counseling metrics. • We introduce a unified evaluation framework that consolidates evaluations from prior work and expands expert assessment to nine aspects of counseling. 2 Related Work Synthetic Counseling Data Generation Due to privacy constraints limiting access to real counseling data, there is growing interest in synthetic counseling dialogue generation. Psych8k ( Liu et al. 2023 ) introduced a framework that generates counselor responses to client questions. However, it is limited to single-turn interactions. To address the need for multi-turn conversations, subsequent works such as SMILE ( Qiu et al. 2024 ) and SoulChat ( Chen et al. 2023 )