Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DO

Rune M. Jacobsen, Samuel Rhys Cox, Carla F. Griggio, Niels van Berkel · 2025

Read Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DO by Rune M. Jacobsen, Samuel Rhys Cox, Carla F. Griggio, Niels van Berkel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08582.txt.

Introduction

Rune Jacobsen, Samuel Rhys Cox, Carla Griggio, and Niels van Berkel published this paper in 2025. It tests whether chatbots can enrich surveys by interviewing respondents the way a skilled human would. Surveys scale well but yield thin answers; interviews run deep but stay small.

The team built four probes from interview theory: descriptive, idiographic, clarifying, and explanatory follow-ups, delivered by a language-model chatbot inside the survey. Sixty-four participants answered across three stages of product research: exploration, requirements, and evaluation. All four probes lifted answer quality, but each shone at a different stage. Clarifying probes, for instance, suit early exploration, while explanatory ones fit evaluation. Participants rated the experience well, with few signs of fatigue. The authors give design guidance for researchers who want interview depth at survey scale. They note the probes add time, so placement should be chosen with care.

Catalog record

CREATORRune M. Jacobsen, Samuel Rhys Cox, Carla F. Griggio, Niels van Berkel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08582

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3714128 ISBN: 979-8-4007-1394-1/25/04 CCS: Human-centered computing User studies CCS: Human-centered computing Empirical studies in HCI Rune M. Jacobsen, Samuel Rhys Cox, Carla F. Griggio, Niels van Berkel Abstract . Surveys are a widespread method for collecting data at scale, but their rigid structure often limits the depth of qualitative insights obtained. While interviews naturally yield richer responses, they are challenging to conduct across diverse locations and large participant pools. To partially bridge this gap, we investigate the potential of using LLM-based chatbots to support qualitative data collection through interview probes embedded in surveys. We assess four theory-based interview probes: descriptive, idiographic, clarifying, and explanatory. Through a split-plot study design ( N = 64), we compare the probes’ impact on response quality and user experience across three key stages of HCI research: exploration , requirements gathering , and evaluation . Our results show that probes facilitate the collection of high-quality survey data, with specific probes proving effective at different research stages. We contribute practical and methodological implications for using chatbots as research tools to enrich qualitative data collection. † † cc-license: by 1. Introduction Online surveys are widely used in Human-Computer Interaction (HCI) research as a data collection method. They can reach a wide population ( Evans and Mathur 2018 ) , are cost-effective ( Olson et al. 2020 ) , and responses are quickly collected ( Couper 2008 ) . HCI research has extensively relied on online surveys to generate insights about users’ attitudes, perceptions, habits, awareness, and experiences with technology ( Müller and Sedley 2015 ) . Online surveys can provide both quantitative and qualitative self-reported data. For example, surveys often include a combination of closed-ended questions with restricted response options, facilitating quantification and comparison ( Lazar et al. 2017 ) , and open-ended questions for accessing richer, more detailed insights through respondents’ own words ( Reja et al. 2003 ; Chen 2017 ; Singer and Couper 2017 ) . However, the quality and richness of the qualitative data collected through online surveys is often more limited than when interacting with participants directly, such as when conducting face-to-face interviews. In surveys, any follow-up questions must be predetermined, potentially missing out on clarifying a vague answer or expanding on an interesting insight. Additionally, participant engagement may be less reliable ( Wright 2017 ; Heerwegh and Loosveldt 2008 ; Roster et al. 2004 ; Fricker et al. 2005 ) , resulting in low-quality ( Reja et al. 2003 ) or nonsensical answers ( Gadiraju et al. 2015 ) , or in omitting responses altogether. Beyond data quality, asking open-ended questions in surveys may cause survey fatigue ( Porter 2004a ) , often ascribed to the cognitive load and the additional time and effort required, leading to survey taking fatigue and participant dropout ( Porter et al. 2004 ) . These limitations motivate the exploration of novel interactive methods for data collection that are capable of preserving the time- and cost-effectiveness provided by online surveys without compromising on the richness and quality of open-ended answers. Recent research has explored the feasibility of collecting qualitative data through chatbots embedded in surveys, substituting open-ended questions with chatbot-based interview probes that interact with participants in a conversational style ( Đuka and Njeguš 2021 ; Xiao et al. 2020b ; Ge et al. 2023 ; Kocielnik et al. 2021 ; Harms and Schmidt 2017 ; Kim et al. 2019 ; Wijenayake et al. 2020 ) . Chatbot-driven data collection in surveys has been shown to enhance participant disclosure ( Tallyn et al. 2018 ; Xiao et al. 2019 ) , improve data quality as compared to the traditional open-ended question format ( Xiao et al. 2020b ; Kim et al. 2019 ) , and been hypothesised to reduce satisficing behaviour ( Wijenayake et al. 2020 ) . However, chatbots often show limitations in understanding the diverse user input in responses to open-ended questions ( DeVault et al. 2014 ) , leading to user disappointment and frustration ( Grudin and Jacques 2019 ) . With the recent surge of large-language models (LLMs), chatbot capabilities have taken a leap in their ability to interpret text and provide a sensible response, proposing a promising direction for utilising chatbots in surveys. We propose that LLM-based chatbots, instructed in conversational techniques based on face-to-face interviews, can facilitate the collection of high-quality qualitative data in surveys. A well-established approach to uncovering in-depth insights involves the use of interview probes, which aim to elicit richer and more nuanced insights based on participants’ prior responses ( Price 2002 ; Gorden 1987 ; Russel 2013 ; Robinson 2023 ) . Incorporating such probing techniques in chatbot-driven survey inquiries presents an opportunity to enhance data quality. However, interview probes can also negatively affect the interviewee’s experience ( Willis 2004 ; Camilla Priede and Farrall 2014 ) . This underscores the importance of carefully considering the survey-takers’ experience when designing chatbots that employ probing questions. In this work, we are interested in exploring chatbot designs that can serve as tools for large-scale surveys to inform three common interview stages of HCI research and practice ( Lazar et al. 2017 , pp. 180-187) : Exploration , Requirements , and Evaluation . To address these research gaps, we set out to answer two research questions: RQ1: How can distinct chatbot-based interview probes support qualitative data collection across different interview stages of HCI research? RQ2: What is the user experience of interacting with a chatbot-based interview probe when completing a survey? To address these research questions, we incorporate and assess four theory-based interviewing probes ( Robinson 2023 ) in a chatbot for collecting high-quality qualitative data through online surveys. We examine the effectiveness of a Descriptive , Idiographic , Clarifying , and Explanatory interview probes in a between-subject setup across the common HCI research interview stages of Exploration , Requirements , and Evaluation ( Lazar et al. 2017 , pp. 180-187) in a within-subjects design. Our analysis focuses on assessing the impact of each interviewing probe on the quality of responses, using Gricean Maxims (a set of communication principles) to measure the informativeness, relevance, specificity, and clarity, as well as the participants’ experience. We contribute an analysis of the effect of interview probes and research interview stages on response quality, as well as participant experience and perception of survey chatbots. Furthermore, we provide concrete recommendations for employing interview probes in HCI studies and outline opportunities and challenges for integrating LLM-based chatbots in surveys. Finally, we provide the source code and implementation details of our chatbot-based interview probes to support other researchers seeking to employ chatbots for data collection in surveys. 2. Background and Related Work 2.1. Information Elicitation through Online Surveys Conducting surveys is one of the primary methods for collecting insights into the opinions, experiences and perceptions of people ( Evans and Mathur 2005 ) . In surveys, researchers and practitioners utilise a series of questions to elicit information on a topic of interest

Read the complete text