Applying Think-Aloud in ICTD: A Case Study of a Chatbot Use by Teachers in Rural Côte d’Ivoire

Vikram Kamath Cannanure, Sharon Wolf, Kaja Jasińska, Timothy X Brown, Amy Ogan · 2025

Read Applying Think-Aloud in ICTD: A Case Study of a Chatbot Use by Teachers in Rural Côte d’Ivoire by Vikram Kamath Cannanure, Sharon Wolf, Kaja Jasińska, Timothy X Brown, Amy Ogan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05840.txt.

Introduction

Vikram Kamath Cannanure and co-authors test a classic usability method, thinking aloud, with teachers using a chatbot in rural Côte d'Ivoire. In the method, users say their thoughts while they work. It is a staple of interface design, but its fit across cultures is untested.

Two weeks after the chatbot launched, twenty teachers took part in guided sessions. The team logged how they navigated, where they erred, and how long tasks took. The sessions surfaced design flaws and features that confused the teachers. The authors reflect on how the method itself had to bend to local norms. Their notes aid teams who build tools for the wider world. A method built in one culture cannot simply be dropped into another.

Catalog record

CREATORVikram Kamath Cannanure, Sharon Wolf, Kaja Jasińska, Timothy X Brown, Amy Ogan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-05840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Applying Think-Aloud in ICTD: A Case Study of a Chatbot Use by Teachers in Rural Côte d’Ivoire Vikram Kamath Cannanure, Sharon Wolf, Kaja Jasińska, Timothy X Brown, Amy Ogan Abstract . Think-alouds are a common HCI usability method where participants verbalize their thoughts while using interfaces. However, their utility in cross-cultural settings, particularly in the Global South, is unclear, where cultural differences impact user interactions. This paper investigates the usability challenges teachers in rural Côte d’Ivoire faced when using a chatbot designed to support an educational program. We conducted think-aloud sessions with 20 teachers two weeks after a chatbot deployment, analyzing their navigation, errors, and time spent on tasks. We discuss our approach and findings that helped us identify usability issues and challenging features for improving the chatbot designs. Our note summarizes our reflections on using think-aloud and contributes to discussions on its culturally sensitive adaptation in the Global South. Conference: Information and Communication Technologies and Development; Dec 09–11, 2020; Nairobi, Kenya CCS: Human-centered computing Empirical studies in HCI CCS: Applied computing Interactive learning environments CCS: Social and professional topics Cultural characteristics 1. Introduction Think-aloud methods are widely used in usability research to gain insights into users’ cognitive processes during task performance ( Ramey et al., 2006 ; Nielsen et al., 2002 ) . This method involves participants verbalizing their thoughts while completing a task, providing researchers access to unobservable mental processes during user interactions ( Charters, 2003 ) . Think-aloud has been applied in various fields in HCI, such as education and qualitative research ( García et al., 2018 ) . Proper design, test procedures, and analysis are integral to using this method ( Nielsen et al., 2002 ) . However, while the think-aloud method offers valuable insights in Western contexts, researchers must remain aware of potential limitations and explore opportunities to adapt this method to different contexts appropriately ( Nielsen et al., 2002 ) . Applying think-aloud techniques in the Global South introduces additional challenges due to cultural differences. Think-Aloud research across Africa, India, and Denmark has shown that traditional Western usability testing methods are less utility in cross-cultural settings ( Clemmensen and Goyal, 2006 ; Oyugi et al., 2008 ) , highlighting variations in personal interaction norms ( Oyugi et al., 2008 ) . Recognizing and addressing these cultural factors is vital when conducting user-centered design testing in diverse contexts ( Perri et al., 2014 ; Dell and Kumar, 2016 ) . The utility of think-aloud tests is often maximized when evaluators and test users share the same cultural background ( Clemmensen and Goyal, 2006 ) , thus requiring the capacity building to train prospective researchers. As HCI4D grows, it is increasingly important to adapt methodologies for cultural relevance to serve under-resourced populations better worldwide ( Dell and Kumar, 2016 ) . Our project describes a think-aloud session for teachers using a chatbot in rural Côte d’Ivoire. Recently, chatbots have become popular in HCI, providing natural language interfaces to digital systems ( Brandtzaeg and Følstad, 2017 ; Cannanure et al., 2020a ; Jain et al., 2018 ) . Think-aloud techniques remain a prominent method to understand chatbot usability and design ( Nielsen et al., 2002 ; Barbosa et al., 2022 ) . Barbosa et al. (2022) found that think-aloud was a favorable method for users using a chatbot, and it helped capture users’ insights over other methods ( Barbosa et al., 2022 ) . As chatbots evolve, further refinement of UX evaluation methods is needed to address new challenges ( Hristidis, 2018 ) . Figure 1. Think-aloud session with a teacher, conducted two weeks after training. The teachers were asked to use the chatbot features as the interviewer recorded their attempts. They were given three attempts per feature, and the entire session was recorded on video. We conducted 20 think-aloud sessions (see Figure 1 ) with rural teachers to observe how they interact with the chatbot. Teachers were asked to navigate the chatbot’s different features (see Table 1 ) two weeks after deployment. Through our work, we hope to contribute our reflections and discussions for adapting think-aloud to the Global South. 2. Methodology The study occurred in a southwestern region of Côte d’Ivoire in May 2021, where French is the official language, though nearly 70 local languages are spoken ( Simons and Fennig, 2017 ) . The main occupation is agricultural, with cocoa and coffee being key income sources for decades ( Kiewisch, 2015 ) . The specific site, RegionM in the Soubré region, is a rural town with most schools located in remote areas facing significant infrastructure challenges, such as limited access to water, electricity, and phone signals, and poor road conditions, which contribute to low literacy rates among students ( Madaio et al., 2019b ; Madaio et al., 2019a ) . The study focused on teachers implementing the newMethod (anonymized) program to improve basic math and French skills among 3rd, 4th, and 5th graders in rural schools. newMethod program groups students by proficiency level, and teachers conduct 45-minute daily sessions of interactive, child-centered activities to enhance learning. After gaining initial success, the Ivorian government was interested in scaling up the program with technology ( Cannanure et al., 2020b ; Cannanure et al., 2022 ) Think-Alouds Sessions Two weeks after the chatbot was introduced, a series of think-aloud sessions were conducted with 20 teachers to evaluate their ability to use the different chatbot features (See Appendix A ). Each teacher was given three attempts to access the feature while verbalizing their thought process. An interviewer observed and recorded their attempts using a survey. For the interviewer, each feature was a survey question with instructions on conducting the think-aloud session, i.e., noting steps taken, success or failure, errors, and time spent. The entire session was video-recorded for further analysis. We also had a follow-up interview with the participants. Data Analysis We consolidated field observation notes and interview transcripts into a single dataset. The interview data was transcribed and translated into English, and we applied thematic analysis ( Charmaz, 2008 ; Strauss and Corbin, 1990 ; Muller, 2014 ; Beyer and Holtzblatt, 1998 ) to identify low-level themes. These themes were synthesized through discussions with the research team and field team to address our research questions. 3. Findings Table 1. The success rate of the think-aloud by feature and teacher. Each row represents a teacher, and each column represents a feature. A green box indicates success and a red box indicates failure. The columns at the end (User Score) indicate the aggregate success percentage for the user. The last row (Feature %) indicates the aggregate success percentage for the feature. Please see the Appendix A for more information Participant Bonjour Survey Goals Browse Tips Manual Ask question Access story Access joke Add story Answer question User score (%) P1 1 1 0 0 0 0 0 0 0 0 0 18.2 P2 1 1 0 1 0 0 1 0 0 0 0 36.4 P3 1 1 0 0 1 1 0 0 0 0 0 36.4 P4 1 1 0 0 1 1 0 1 0 1 0 54.5 P5 1 1 1 1 1 0 1 0 0 0 0 54.5 P6 1 1 0 1

Read the complete text