Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction

Séverin, Yanis, Shashi, Joonas, Sergio, Pawel, Juan Ignacio, Petr, Hervé, Ricard · 2026

Read Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction by Séverin, Yanis, Shashi, Joonas, Sergio, Pawel, Juan Ignacio, Petr, Hervé, Ricard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06373.txt.

Introduction

Baroudi and his team study Hindi English code switched doctor patient conversations from DISPLACE-M, and DASHA, which chains diarization, speaker tagged transcription, and LLM based extraction. A neural system with vector clustering on a multilingual encoder cuts error rate to 7.76 percent. A Qwen3 ASR model adapted to Hindi with Devanagari normalization cuts time constrained word error rate a lot, with further gains from optional LLM correction. Such gains lift that work only with adapted ASR, and DASHA ranked first among 25 DISPLACE-M participants.

Catalog record

CREATORSéverin, Yanis, Shashi, Joonas, Sergio, Pawel, Juan Ignacio, Petr, Hervé, Ricard
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24341
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06373

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Doctor or Patient? Synergizing Diarization and ASR for Code-Switched Hinglish Medical Conditions Extraction Séverin, Yanis, Shashi, Joonas, Sergio, Pawel, Juan Ignacio, Petr, Hervé, Ricard Abstract Extracting patient medical conditions from code-switched clinical spoken dialogues is challenging due to rapid turn-taking and highly overlapped speech. We present a robust system evaluated on the DISPLACE-M dataset of real-world Hinglish medical conversations. We propose an End-to-End Neural Diarization with Vector Clustering approach (EEND-VC) to accurately resolve dense and speaker overlaps in Doctor-Patient Conversations (DoPaCo). For transcription, we adapt a Qwen3 ASR model via domain-specific fine-tuning, Devanagari script normalization, and dialogue-level LLM error correction, achieving an 18.59% tcpWER. We benchmark open and proprietary LLMs on medical condition extraction, comparing our text-based cascade system against a multimodal End-to-End (E2E) audio framework. While proprietary E2E models set the performance ceiling, our open cascaded architecture is highly competitive, as it achieved first place out of 25 participants 1 1 1 CodaBench Leaderboard in the DISPLACE-M challenge. All implementations are publicly released 2 2 2 GitHub will be released after anonymity period . Baroudi Labrak Kumar Kalda Burdisso Cyrta Alvarez-Trejos Motlíček Bredin Marxer keywords speaker diarization, speaker-attributed automatic speech recognition, medical condition extraction, code-switching † † address: 1 Université de Toulon, Aix Marseille Univ, CNRS, LIS, Marseille, France 2 Idiap Research Institute, Martigny, Switzerland 3 pyannoteAI, Toulouse, France 4 MILA, Montréal, Canada 5 CNRS, ILLS, France 6 Stenograf 7 EPFL, Lausanne, Switzerland † † email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 1 Introduction Extracting patient medical conditions from real-world medical dialogues is a complex task, particularly in recordings featuring noisy far-field recordings, and spontaneous code-mixed speech. This challenge is exemplified in multilingual interactions combining Hindi and English (Hinglish), where linguistic fluidity complicates standard extraction pipelines. Prior work in medical dialogue systems [ 1 , 2 ] has largely focused on monolingual English interactions [ 3 ] , often relying on large-scale proprietary datasets [ 1 ] and models. While significant progress has been made in automated clinical note generation [ 4 , 5 , 6 ] , these approaches typically struggle when applied to low-resource languages [ 7 ] or code-switched speech, such as Hinglish [ 8 , 9 ] . Given the scarcity of medical conversational speech resources, we conduct our study on the DISPLACE-M dataset [ 10 ] , which comprises approximately 35 hours (including 25 hours for training) of unseen real primary-healthcare interactions recorded between community health workers (ASHAs) and patients in rural India. These fully de-identified recordings reflect the difficulties posed by multilingual code-mixed conversational speech in noisy environments and present a low risk of data contamination in current models. Beyond linguistic complexities, such real-world interactions are characterized by spontaneous acoustic dynamics that complicate speaker diarization. On the other hand, recent hybrid architectures [ 11 , 12 ] try to solve Speaker-Attributed Automatic Speech Recognition ( SA-ASR ) utilizing ECAPA-TDNN [ 13 ] for role diarization to distinguish patient and doctor. Their initiative is an non-optimal approach for highly interactive environments and their reliance on the assumption of sequential speech. To overcome these limitations, our method leverages the End-to-End Neural Diarization with Vector Clustering (EEND-VC) approach [ 14 ] to natively model simultaneous activations and robustly resolve speaker overlap. On top of that, our SA-ASR system employs an ASR module to condition transcriptions exclusively on the active speech of the patient and the doctor given by the diarization module outputs. This simple cascade approach effectively prevents the system from transcribing environmental noise or irrelevant background speech. By integrating these components into a clear cascade architecture (Diarization \rightarrow SA-ASR \rightarrow Extraction), we ensure robust speaker tracking even in dense interactions. This modularity provides significant architectural convenience, as each functional block can be independently replaced as state-of-the-art models evolve. Figure 1: Visualization of the proposed cascade pipeline. Conversations involve code-switching Hinglish turns between patient and doctor. English words (displayed in blue color) are spelled in Devanagari script. Our main contributions are: (1) a robust diarization system tailored for highly overlapping speech aligned with Doctor-Patient Conversations (DoPaCo) use cases; (2) effective adaptation of an ASR approach to Hindi & medical domains, while also featuring the canonical normalization of the Devanagari script; (3) a straightforward End-To-End solution to medical conditions extraction; and (4) overall, a highly competitive pipeline that achieved first place out of 25 participants in DISPLACE-M challenge as team ILIP {}^{\hrefhttps://www.codabench.org/competitions/13833/?secret_key=1b714e64-0f0d-4e0f-8a3c-be9b3d10f00c} and a public release of individual implementations to ensure reproducibility. 2 Methodology 2.1 Dataset Overview To train and adapt our models to the unique challenges of Hinglish clinical dialogues, we established a comprehensive data pipeline using the DISPLACE-M corpus. We concatenated two in-domain development sets, DEV1 and DEV2, into a unified DEV set. This combined dataset comprises 125 dialogues (78 from DEV1 and 47 from DEV2) and totals 30.1k conversational turns, yielding an overall duration of 24.5 hours. Individual dialogues range from 43 to 565 turns, averaging 241.2 turns and approximately 11 minutes and 47 seconds per session. In addition to the development data, we utilize the blind eval set (EVAL1) totaling 10.13 hours to evaluate all our systems. The audio data is distributed across 84.30% single-speaker speech, 11.35% silence, and 4.35% overlapping speech. This level of overlap is comparable to established benchmarks such as the VoxConverse diarization challenge [ 15 ] , where average overlap typically hovers around a similar 34̃%, presenting a significant challenge for precise speaker attribution. These dyadic recordings are further characterized by linguistic complexity due to frequent code-switching between Hindi and English, particularly regarding specialized medical terminology and the contrasting semantic density between doctor and patient speech. 2.2 Speaker Diarization While the DiariZen [ 16 ] system serves as a strong baseline for the diarization module, it can be further optimized for Hindi and conversational content. A key requirement for this adaptation is a robust multilingual conversational speech encoder. DiariZen uses WavLM-Base [ 17 ] , a model primarily pretrained on English-language data and mostly on non-conversational datasets (i.e. Librispeech [ 18 ] ). In contrast, studies in [ 19 , 20 ] demonstrate that w2v-bert2.0 [ 21 , 22 ] , is a stronger alternative for diarization tasks on Hindi conversations. Trained on 143 languages and 4.5 million hours of data from diverse domains, w2v-bert2.0 exhibits strong generalization capabilities, particularly for unseen data [ 20 ] . We adopt w2v-bert2.0 as our backbone, followed by the EEND hybrid approach, using the powerset speaker segmentation model [ 23 , 24 ] . Following the extraction of frame-level features, we evaluate the performance of two distinct neural architectures for the context network: LSTMs and Mamba layers [ 25 ] . While established systems such as DiariZen [ 16 ] utilize Conformers [ 26 ] , recent advancements in EEND segmentation [ 27 ] suggest that Mamba layers offer superior potential; in contrast to the quadratic computational overhead inherent in Conformer-based attention, Mamba leverages a Selective State Space Model (SSM) to capture extended speaker context with linear complexity. A simple linear layer on the output of the previous context layer produces frame-wise diarization. Because our focus is solely on predicting patient and doctor speech activity, we hypothesize that restricting the model to a 2-speaker output would improve performance,

Read the complete text