A Fully Generative Motivational Interviewing Counsellor Chatbot for Moving Smokers Towards the Decision to Quit
Zafarullah Mahmood * Soliman Ali * Jiading Zhu * Mohamed Abdelwahab * Michelle Yu Collins * Sihan Chen * Yi Cheng Zhao *, Jodi Wolff, Osnat Melamed, Nadia Minian, Marta Maslej, Carolynne CooperMatt Ratto *, Peter Selby, Jonathan Rose * University of Toronto · 2025
Read A Fully Generative Motivational Interviewing Counsellor Chatbot for Moving Smokers Towards the Decision to Quit by Zafarullah Mahmood * Soliman Ali * Jiading Zhu * Mohamed Abdelwahab * Michelle Yu Collins * Sihan Chen * Yi Cheng Zhao *, Jodi Wolff, Osnat Melamed, Nadia Minian, Marta Maslej, Carolynne CooperMatt Ratto *, Peter Selby, Jonathan Rose * University of Toronto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17362.txt.
Introduction
Mahmood and co-authors build a fully generative chatbot that counsels smokers toward quitting, using a therapy style called motivational interviewing. The style builds a person's own reasons to change rather than lecturing. Their 2025 chatbot pairs a leading language model with that method, shaped alongside clinicians who teach the approach.
In a test with 106 smokers, confidence in quitting rose by an average of 1.7 points on a ten-point scale one week later. Automatic scoring found the bot followed the method in ninety-eight percent of its turns, above human counselor rates. Smokers rated its empathy well, though below human counselors, and their own words showed strong motivation to change. The authors see promise in automated talk therapy.
Catalog record
| CREATOR | Zafarullah Mahmood * Soliman Ali * Jiading Zhu * Mohamed Abdelwahab * Michelle Yu Collins * Sihan Chen * Yi Cheng Zhao *, Jodi Wolff, Osnat Melamed, Nadia Minian, Marta Maslej, Carolynne CooperMatt Ratto *, Peter Selby, Jonathan Rose * University of Toronto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Fully Generative Motivational Interviewing Counsellor Chatbot for Moving Smokers Towards the Decision to Quit Zafarullah Mahmood * Soliman Ali * Jiading Zhu * Mohamed Abdelwahab * Michelle Yu Collins * Sihan Chen * Yi Cheng Zhao *, Jodi Wolff, Osnat Melamed, Nadia Minian, Marta Maslej, Carolynne CooperMatt Ratto *, Peter Selby, Jonathan Rose * University of Toronto Abstract The conversational capabilities of Large Language Models (LLMs) suggest that they may be able to perform as automated talk therapists. It is crucial to know if these systems would be effective and adhere to known standards. We present a counsellor chatbot that focuses on motivating tobacco smokers to quit smoking. It uses a state-of-the-art LLM and a widely applied therapeutic approach called Motivational Interviewing (MI), and was evolved in collaboration with clinician-scientists with expertise in MI. We also describe and validate an automated assessment of both the chatbot’s adherence to MI and client responses. The chatbot was tested on 106 participants, and their confidence that they could succeed in quitting smoking was measured before the conversation and one week later. Participants’ confidence increased by an average of 1.7 on a 0-10 scale. The automated assessment of the chatbot showed adherence to MI standards in 98% of utterances, higher than human counsellors. The chatbot scored well on a participant-reported metric of perceived empathy but lower than typical human counsellors. The participants’ language also indicated a good level of motivation to change, a key goal in MI. These results suggest that automation of talk therapy with a modern LLM has promise. $\mathsection$ $\mathsection$ footnotetext: Corresponding author: [email protected] 1 Introduction The remarkable rise in the capability of large language models (LLMs) gives hope that they could be used to provide many kinds of mental health talk therapy. Indeed, one can simply ask for such help from an online LLM and possibly receive good help ( Siddals et al., 2024 ) . Since this is a medical intervention, it should be grounded in evidence that shows its effectiveness. Our goal is to automate a specific type of talk therapy focusing on the problem of tobacco addiction with the specific goal of moving ambivalent smokers towards the decision to quit. Ambivalent smokers know that smoking is bad for them but continue smoking because of its positive effects and because they don’t spend much time contemplating their smoking behaviour ( Miller, 1983 ; Rollnick et al., 1997 ; Miller and Rollnick, 2023 ) . More than 50% of all smokers are in this ambivalent state Babb et al. (2017) , and so moving even a small fraction of these towards the decision to quit could have a major impact. The Motivational Interviewing (MI) talk therapy approach ( Miller and Rollnick, 2023 ) is often employed by counsellors to guide smokers away from their ambivalent state towards the decision to quit. This decision is a key precursor for any successful attempt to quit ( West and Sohal, 2006 ) . There has been significant activity in recent years on automating talk therapy in many domains, including the use of MI to help in smoking cessation ( Steenstra et al., 2024 ; Basar et al., 2024 ; Welivita and Pu, 2023 ; Brown et al., 2023 ) . Brown et al. (2023) , the predecessor of the present work, developed MIBot v5.2 which showed that a partially scripted and partially generative chatbot could significantly change smokers’ readiness to quit. However, scripting with limited generation restricts the natural flow of conversation, thereby preventing full utilization of MI elements. Steenstra et al. (2024) show the effectiveness of a fully-generative chatbot focused on alcohol use. As well, more complete MI administered by human counsellors has shown a much greater impact ( Boudreaux et al., 2012 ) . This, together with the potential availability of always-accessible, lower-cost counselling, forms the motivation for this work. In this paper, we describe the design and measurement of a single, large prompt of a state-of-the-art LLM-based chatbot called MIBot 1 1 1 This paper describes MIBot v6.3A and compares it with our previous work, MIBot v5.2 ( Brown et al., 2023 ) . Our group’s broader goal is to iteratively develop MI-based chatbots for smoking cessation. See Appendix G for a comprehensive list of all previous MI chatbot iterations. Unless otherwise noted, MIBot refers to MIBot v6.3A . . A key to our approach is that expert MI clinicians and researchers participated in designing the prompt and evaluating the chatbot. We iteratively evolved the prompt with the help of MI experts, LLM-simulated smokers, and humans role-playing as smokers. MIBot was then tested on smokers recruited online (for pay) to measure both the effect on their confidence to quit and the quality of the conversations in four ways: 1. The participants’ readiness to quit through a widely used readiness ruler ( Boudreaux et al., 2012 ) before the conversation and one week later. The difference between these two measurements is our primary metric of effectiveness. 2. A rating of the perceived empathy of the chatbot on the CARE scale ( Mercer et al., 2004 ) , which is widely used to assess the quality of the clinician-patient interaction and clinician empathy. 3. A measurement of how well the counsellor’s utterances adhere to the standards of MI based on the Motivational Interviewing Skill Code ( MISC ) ( Houck et al., 2010 ) . 4. The percentage of client utterances that reflect their motivation to change their smoking behaviour as a portion of the total number of utterances that reflect either change or the sustaining of their behaviour — also based on MISC. The key contributions of this paper are: 1. An expert-informed chatbot that performs fully generative MI counselling. 2. Measurements of effectiveness on human smokers. 3. A validated automated system to measure the adherence of counsellor chatbot utterances to the precepts of MI. 4. A validated automated measurement of the effect of the chatbot on the client’s motivation through analysis of their language. 5. A dataset of the transcripts of 106 chatbot-human conversations together with measured outcomes of effectiveness, perceived empathy, and utterance-level MISC annotations 2 2 2 https://github.com/cimhasgithub/MIBOT_ACL2025 . This paper is organized as follows: the next section describes prior work in the area of automated MI using therapeutic chatbots (and their evaluation). Section 3 describes the clinician-informed iterative design of MIBot . Section 4 discusses the methods of measurement and recruitment of human smokers. Section 5 presents the results and discussion, and Section 6 concludes. Figure 1: Overview of the MIBot system and observer agents. 2 Related work 2.1 MI-Based Chatbots Various approaches to building MI-based chatbots have been explored in the past. Some have relied on pre-scripted or structured responses directed by rule-based logic ( Park et al., 2019 ; Olafsson et al., 2019 ; Samrose and Hoque, 2022 ) . Structured responses ensure adherence to MI principles, but lack the flexibility to adapt to user input. Others have incorporated natural language understanding (NLU) units to help structure the conversation flow ( Almusharraf et al., 2020 ; He et al., 2022 ; Saiyed et al., 2022 ; Leeuwis and He, 2023 ; He et al., 2024 ) , allowing for some adaptability. Following their