Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization

Mohammed Aman Bhuiyan, Md Sazzad Hossain Adib, Samiul Basir Bhuiyan, Amit Chakraborty · 2026

Read Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization by Mohammed Aman Bhuiyan, Md Sazzad Hossain Adib, Samiul Basir Bhuiyan, Amit Chakraborty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08214.txt.

Introduction

Mohammed Aman Bhuiyan and colleagues build speech tools for Bangla, a language where recognition and speaker diarization still lag. Long recordings, varied acoustics, and wide speaker differences make both tasks hard, and off-the-shelf models are tuned for other languages.

For recognition, the team fine-tunes a regional Whisper model on about fifteen thousand curated Bangla segments, with heavy augmentation including noise, reverb, echo, clipping, and pitch and time changes. For diarization, they fine-tune the pyannote segmentation model and swap it into the community pipeline. The paired systems target practical long-form Bangla transcription that also says who spoke when.

Catalog record

CREATORMohammed Aman Bhuiyan, Md Sazzad Hossain Adib, Samiul Basir Bhuiyan, Amit Chakraborty
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08214

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization Mohammed Aman Bhuiyan, Md Sazzad Hossain Adib, Samiul Basir Bhuiyan, Amit Chakraborty Abstract Automatic Speech Recognition (ASR) and speaker diarization in Bangla remain challenging due to long-form recordings, diverse acoustic conditions, and significant speaker variability. This work addresses these two core tasks in Bangla spoken language understanding by developing robust systems for long-form ASR and speaker diarization. For ASR (Problem 1), we fine-tune the tugstugi_bengaliai-regional-asr_whisper-medium model on a custom-curated dataset of approximately 15,000 chunked and aligned Bangla audio segments, employing full-weight training with extensive data augmentation including noise injection, reverb simulation, echo, clipping distortion, and pitch/time perturbation. For speaker diarization (Problem 2), we fine-tune the pyannote/segmentation-3.0 model using PyTorch Lightning on the competition’s annotated diarization dataset, swapping the fine-tuned segmentation backbone into the pyannote/speaker-diarization-community-1 pipeline while retaining the pretrained speaker embedding and clustering components. Our ASR system achieves a Word Error Rate (WER) of 24.41 %, while our diarization system achieves a Diarization Error Rate (DER) of 23.92 %, both evaluated on the test set, demonstrating notable improvements over the respective pretrained baselines. We describe our complete pipeline, including data preprocessing, text normalization, audio augmentation, training strategies, inference optimization, and post-processing for both tasks. The corresponding code has been made publicly available at https://github.com/sazzadadib/BitwiseMind_DL_Sprint_4.0 . I Introduction Bangla (Bengali), spoken by over 230 million people worldwide, is one of the most widely spoken languages globally, yet it remains significantly under-resourced in the domain of speech technology. Long-form audio understanding, which includes both accurate transcription and speaker attribution, is critical for applications ranging from media indexing and meeting summarization to accessibility tools and legal or forensic audio analysis [ 1 ] . Despite major progress in end to end ASR and large pretrained models, most state of the art systems remain optimized for high resource languages like English, resulting in higher error rates for Bangla, especially in spontaneous, multi speaker, and diverse domain audio. Addressing this gap requires carefully curated datasets, robust modeling approaches, and evaluation strategies tailored to the unique linguistic and acoustic characteristics of Bangla. DL Sprint 4.0 poses two challenging problems on Bangla long-form audio: Fig. 1: Comparison of pretrained baselines and the proposed system. Our method improves both WER and DER. • Problem 1 — Bangla Long-Form ASR: Transcribe extended Bangla speech recordings into text, evaluated using Word Error Rate (WER). • Problem 2 — Speaker Diarization: Determine “who spoke when” by producing speaker-labelled time segments, evaluated using Diarization Error Rate (DER). Both problems present unique challenges for Bangla. The ASR task must handle dialectal variation, conversational speech, environmental noise, and the morphological complexity of the Bangla script. The diarization task must accurately segment long recordings with potentially overlapping speakers and varying acoustic conditions. In this paper, we present our approach to both problems. For ASR, we leverage OpenAI’s Whisper architecture [ 1 ] , specifically the medium variant pre-trained on Bengali by Tugstugi [ 8 ] , and perform fine-tuning with a comprehensive data augmentation pipeline. For diarization, we adopt PyAnnote.Audio’s segmentation-3.0 model [ 4 , 3 ] and fine-tune it on the competition dataset, integrating the fine-tuned weights into the full diarization pipeline. Our key contributions are: 1. A complete data preprocessing pipeline for Bangla long-form audio, including automated chunking with ASR-guided ground-truth alignment using fuzzy matching. 2. A comprehensive audio augmentation strategy combining noise injection, room reverb simulation, multi-tap echo, clipping distortion, bandpass filtering, pitch shifting, and time stretching to improve model robustness. Bangla-specific text normalization including English numeral-to-Bangla word conversion and non-Bengali character removal. 3. Longform audio ASR model fine-tuning, followed by domain-adaptive fine-tuning of PyAnnote’s segmentation model using checkpoint-based training and segmentation-swap inference for speaker diarization. 4. Post-processing techniques including hallucination removal for ASR and short-segment filtering for diarization. Fig. 2: End-to-end pipeline for Bangla Long-Form Speech Recognition, using fine-tuned Whisper-medium(BengaliAI) ASR model. The framework demonstrates data preprocessing, dataset chunking, model fine-tuning, and inference stages for transcribing long-form speech to standard Bangla transcription orthography. II Related Work II-A Automatic Speech Recognition Modern ASR systems have shifted from traditional Hidden Markov Model (HMM) and Connectionist Temporal Classification (CTC) approaches toward end-to-end encoder-decoder architectures. OpenAI’s Whisper [ 1 ] represents a significant advance in multilingual ASR, trained on 680,000 hours of weakly supervised data spanning 99 languages. Whisper uses a Transformer-based encoder-decoder architecture that converts log-Mel spectrograms to text in an autoregressive fashion. While Whisper supports Bangla, its performance on long-form conversational Bangla speech remains suboptimal without domain-specific adaptation. BengaliAI has released several fine-tuned Whisper variants for regional Bengali ASR [ 2 ] , which serve as strong starting points for further fine-tuning. Data augmentation for speech has been extensively studied, including SpecAugment [ 5 ] , noise injection, and room impulse response (RIR) simulation [ 6 ] , all of which we adapt for our pipeline. II-B Speaker Diarization Speaker diarization has evolved from clustering-based methods (e.g., agglomerative hierarchical clustering on speaker embeddings) to end-to-end neural approaches. PyAnnote.Audio [ 3 , 4 ] provides a modular pipeline that combines neural speaker segmentation with speaker embedding extraction and clustering. The segmentation-3.0 model uses a PyanNet architecture to produce frame-level speaker activity probabilities; these are then paired with speaker embeddings to perform full diarization. Recent work on low-resource diarization has shown that fine-tuning pretrained segmentation models on target-domain data can significantly reduce DER [ 4 ] . Our approach follows this paradigm, adapting the multilingual pretrained model to the specific acoustic and speaker characteristics of Bangla conversational audio. III Methodology III-A Problem 1 — Bangla Long-Form ASR Our ASR pipeline in Figure 2 consists of four main stages: (1) data preprocessing and chunking, (2) text normalization and cleaning, (3) model fine-tuning, and (4) inference with post-processing. III-A 1 Data Preprocessing and Audio Chunking Long-form audio recordings (varying from several minutes to over an hour) are first chunked into segments of at most 25 seconds, the practical input limit for Whisper’s encoder. We employ a fixed-length non-overlapping chunking strategy: 1. Each training audio file is loaded at 16 kHz. 2. The audio is divided into non-overlapping 25-second segments. 3. Silent or near-silent trailing fragments (below 0.5 s) are discarded. 4. Each chunk is transcribed using the pretrained BengaliAI Whisper model to obtain an ASR hypothesis. To obtain ground-truth labels for each chunk, we implement a sequential fuzzy alignment algorithm. The full ground-truth transcript is split into words, and a sequential pointer is maintained across the corpus. For each ASR-transcribed chunk, a bidirectional search window ( \pm 5 words) and span-length variation ( \pm 3 words) are used to find the best-matching ground-truth span via RapidFuzz ratio scoring. This ensures reading-order preservation while accommodating minor ASR insertions and deletions. III-A 2 Text Normalization and Cleaning We apply several Bangla-specific normalization steps to ensure consistency and linguistic correctness in the transcripts. First, all English-digit sequences are converted into their Bangla word equivalents using the num2words library, with four-digit numbers in the range 1000–2099 specifically interpreted as calendar years. Next, characters outside the Bengali Unicode block (U+0980–U+09FF) are removed, while allowing digits, whitespace, and common punctuation marks. Finally, multiple consecutive whitespace characters are collapsed into a single space to maintain clean and uniform text formatting. III-A

Read the complete text