"Mm, Wat?" Detecting Other-initiated Repair Requests in Dialogue
Anh Ngo, Nicolas Rollet, Catherine Pelachaud, Chloé Clavel · 2025
Read "Mm, Wat?" Detecting Other-initiated Repair Requests in Dialogue by Anh Ngo, Nicolas Rollet, Catherine Pelachaud, Chloé Clavel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24628.txt.
Introduction
Anh Ngo, Nicolas Rollet, Catherine Pelachaud, and Chloe Clavel study small signs that a talk has gone wrong. In Dutch chats, one speaker may hint that they missed what was said, and ask the other to fix it. Chat bots often miss these hints and lose the user.
The team built a model that joins word cues with tone of voice cues. Tone cues added clear gains over text and sound scores used on their own. Their work also shows how the two cue types work as a pair. Next steps may add face cues and tests in more tongues.
Catalog record
| CREATOR | Anh Ngo, Nicolas Rollet, Catherine Pelachaud, Chloé Clavel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24628
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
"Mm, Wat?" Detecting Other-initiated Repair Requests in Dialogue Anh Ngo, Nicolas Rollet, Catherine Pelachaud, Chloé Clavel Abstract Maintaining mutual understanding is a key component in human-human conversation to avoid conversation breakdowns, in which repair, particularly Other-Initiated Repair (OIR, when one speaker signals trouble and prompts the other to resolve), plays a vital role. However, Conversational Agents (CAs) still fail to recognize user repair initiation, leading to breakdowns or disengagement. This work proposes a multimodal model to automatically detect repair initiation in Dutch dialogues by integrating linguistic and prosodic features grounded in Conversation Analysis. The results show that prosodic cues complement linguistic features and significantly improve the results of pretrained text and audio embeddings, offering insights into how different features interact. Future directions include incorporating visual cues, exploring multilingual and cross-context corpora to assess the robustness and generalizability. 1 Introduction Conversational agents (CAs), software systems that interact with users using natural language in written or spoken form, are increasingly being used in multiple domains such as commerce, healthcare, and education Allouch et al. 2021 . While maintaining smooth communication is crucial in these settings, current state-of-the-art (SOTA) CAs still struggle to handle conversational breakdowns. Unlike humans, who rely on conversational repair to resolve issues like mishearing or misunderstanding Schegloff et al. 1977 ; Schegloff 2000 , CAs’ repair capabilities remain limited and incomplete. Repair refers to the interactional effort by which participants suspend the ongoing talk to address potential trouble, which can be categorized by who initiates and who resolves it: the speaker of the trouble (self) or the co-participant (other) ( Schegloff 2000 ) . This work focuses on Other-initiated Self-repair , in short, Other-initiated Repair (OIR) , where the talk of a speaker is treated as problematic by a co-participant via repair initiation, and the original speaker resolves it, as illustrated in Figure 1 . Current CAs handle repairs in a limited fashion that mainly support self-initiated repair by the agent (e.g., the agent asks users to repeat what they said) Li et al. 2020 ; Cuadra et al. 2021 ; Ashktorab et al. 2019 or rely on user self-correction when users realize troubles and clarify their own intent (e.g., saying "no, I mean…") Balaraman et al. 2023 . However, CAs struggle to recognize when users signal trouble with the agent’s utterances (other-initiated) and fail to provide appropriate repair (self-repaired), while effective communication requires bidirectional repair capabilities Moore et al. 2024 . Supporting this, Gehle et al. 2014 found that robots failing to resolve communication issues quickly caused user disengagement, while van Arkel et al. 2020 showed that basic OIR mechanisms improve communication success and reduce computational and interaction costs compared to relying on pragmatic reasoning. Figure 1: Other-initiated Repair (OIR) sequence example from Rasenberg et al. 2022 , English translated: repair initiation (green) signals trouble of ambiguous object reference disc with candidate understanding horizontally , confirmed by repair solution yes horizontally . Modeling OIR strategies on CAs that recognize user - initiated repair first requires robust automatic repair initiation detection in human-human interaction. Previous work has established foundations for text - based approaches, training with English corpora, and relying on lexical cues Höhn 2017 ; Purver et al. 2018 ; Alloatti et al. 2024 . However, prosodic cues tend to be more cross-linguistically stable than surface forms Dingemanse and Enfield 2015 ; Benjamin 2013 ; Walker and Benjamin 2017 , and can provide valuable insight into the pragmatic functions of expressions like the interjection "huh". Building upon text-based methods, this work focuses on spoken dialogue interaction, where prosodic cues provide additional signals for repair initiation detection that may be missed by text-only models trained on transcriptions. Finally, understanding the OIR sequence also requires examining the local sequential environment of the surrounding turns, which we term "dialogue micro context", based on Schegloff 1987 ’s work on local interactional organization. These gaps motivate our main research question: What are the verbal and prosodic indicators of repair initiation in OIR sequences and how can we model them? To address this, we analyze OIR sequences in a Dutch task-oriented corpus, focusing on text and audio patterns where one speaker initiates repair. Drawing on Conversation Analysis literature, we introduce feature sets and a computational model to detect such requests. Our contributions are in two folds: (1) a novel multimodal model for detecting repair initiations in OIR sequences that integrates linguistic and prosodic features extracted automatically based on the literature, advancing beyond text- or audio-only approaches; (2) provide insights into how linguistic and prosodic features interact and contribute in detection performance, grounded in Conversation Analysis, and what causes model misclassifications. The remainings of this paper is structured as follows: Section 2 reviews SOTA computational models for OIR detection and related dialogue understanding tasks. Section 3 provides the used OIR coding schema and typology, and Section 4 details our approach, including linguistic and prosodic feature design. Section 1 presents our experiment details and results, followed by error analysis in Section 6 . 2 Related Work An early approach to automatic OIR detection was proposed by Höhn 2017 , with a pattern-based chatbot handling user-initiated repair in text chats between native and non-native German speakers. Purver et al. 2018 extended this by training a supervised classifier using turn-level features in English, including lexical, syntactic, and semantic parallelism between turns. More recently, Alloatti et al. 2024 introduced a hierarchical tag-based system for annotating repair strategies in Italian task-oriented dialogue, distinguishing between utterance-specific and context-dependent functions. Closely related, Garí Soler et al. 2025 ’s recent work introduced and investigated the task of automatically detecting word meaning negotiation indicators, where speakers signal a need to clarify or challenge word meanings, a phenomenon that can be seen as a specific form of repair initiation. Although direct research on OIR detection is still limited, advances in related dialogue understanding tasks provide promising foundations for our work. Miah et al. 2024 combined pretrained audio (Wav2Vec2) and text (RoBERTa) embeddings to detect dialogue breakdowns in healthcare calls. Similarly, Huang et al. 2023 used BERT, Wav2Vec2.0, and Faster R-CNN for intent classification, introducing multimodal fusion with attention-based gating to balance modality contributions and reduce noise. Saha et al. 2020 proposed a multimodal, multitask network jointly modeling dialogue acts and emotions using attention mechanisms. More recently, high-performing but more opaque and resource-intensive approaches have emerged, such as Mohapatra et al. 2024 showed that larger LLMs outperform smaller ones on tasks like repair and anaphora resolution, albeit with higher computational cost and latency. Despite robust performance, recent largest models remain difficult to interpret due to their black-box nature and multimodal fusion complexity Jain et al. 2024 . To address this gap, we propose a computational model for repair initiation detection in Dutch spoken dialogue that fuses pretrained text and audio embeddings with linguistic and prosodic features grounded in Conversation Analysis . The model also integrates a multihead attention mechanism to weigh and capture nonlinear relationships across modalities, allowing our model to keep the strengths of multimodal deep learning while offering insight from linguistic and prosodic features to understand their interaction and impact towards model’s decision. Figure 2: OIR sequence organization between 2 speakers A (green) and B (red): (a) Minimal; (b) Non-minimal 3 OIR Coding Schema