An Investigation Into Various Approaches For Bengali Long-Form Speech Transcription and Bengali Speaker Diarization
Epshita Jahan, Khandoker Md Tanjinul Islam, Pritom Biswas, Tafsir Al Nafin · 2026
Read An Investigation Into Various Approaches For Bengali Long-Form Speech Transcription and Bengali Speaker Diarization by Epshita Jahan, Khandoker Md Tanjinul Islam, Pritom Biswas, Tafsir Al Nafin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03158.txt.
Introduction
Epshita Jahan, Khandoker Md Tanjinul Islam, Pritom Biswas, and Tafsir Al Nafin build speech tools for Bengali, a language most systems underserve. Their 2026 paper, developed for two Kaggle competitions, tackles hour-long recordings: transcribe what was said and label who spoke when. Both jobs are hard in noisy, varied conditions.
For transcription they fine-tuned the Whisper model on Bengali data, then added chunking, noise cleaning, and post-processing fixes. For speaker separation they combined an existing diarization pipeline with a custom-trained segmentation model and a tuned second pass. The scores placed respectably on both public and private leaderboards. The team argues targeted tuning and smart data use can close the inclusivity gap for South Asian languages.
Catalog record
| CREATOR | Epshita Jahan, Khandoker Md Tanjinul Islam, Pritom Biswas, Tafsir Al Nafin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17181 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Investigation Into Various Approaches For Bengali Long-Form Speech Transcription and Bengali Speaker Diarization Epshita Jahan, Khandoker Md Tanjinul Islam, Pritom Biswas, Tafsir Al Nafin Abstract Bengali remains a low-resource language in speech technology, especially for complex tasks like long-form transcription and speaker diarization. This paper presents a multi-stage approach developed for the “DL Sprint 4.0 — Bengali Long-Form Speech Recognition” and “DL Sprint 4.0 — Bengali Speaker Diarization” [ 1 ] on Kaggle, addressing the challenges of “who spoke when/what” in hour-long recordings. We implemented Whisper Medium fine-tuned on Bengali data (bengaliAI/tugstugi_bengaliai-asr_whisper-medium) for transcription and integrated pyannote/speaker-diarization-community-1 with our custom-trained segmentation model to handle diverse and noisy acoustic environments. Using a two-pass method with hyperparameter tuning, we achieved a DER of 0.27 on the private leaderboard and 0.19 on the public one. For transcription, chunking, background noise cleaning, and algorithmic post-processing yielded a WER of 0.38 on the private leaderboard. These results show that targeted tuning and strategic data utilization can significantly improve AI inclusivity for South Asian languages. All relevant codes are available at: https://github.com/Short-Potatoes/Bengali-long-form-transcription-and-diarization.git I Introduction Recent advances in speech recognition have been driven by large-scale datasets and powerful models, yet low-resource languages like Bengali have not fully benefited. This study addresses that gap by proposing robust methodologies for long-form transcription and speaker diarization tailored to Bengali. Long-form transcription is challenging because systems like Whisper impose a 30-second input limit, requiring segmentation of lengthy audio. To overcome this, we used voice activity detection (VAD) to identify speech regions, applied Whisper for transcription, and aligned outputs with post-processing to remove redundancies. Our dataset consisted of hour-long recordings with transcriptions, underscoring the need for effective strategies. For speaker diarization, we fine-tuned Pyannote’s segmentation model within its pipeline. A two-pass approach—coarse segmentation followed by refined analysis—improved accuracy. Evaluation was based on long-form audio with ground-truth speaker annotations in JSON format. In summary, combining segmentation-based preprocessing, Whisper-based ASR, and fine-tuned diarization models significantly enhances speech recognition for Bengali and other low-resource languages. II Methodology II-A Model Selection for Speaker Diarization Conversational Bangla diarization presents challenges including speaker overlap, acoustic variability, and unconstrained speaker counts. We evaluated three contemporary diarization frameworks: Pyannote Community [ 2 ] , NVIDIA NeMo Sortformer [ 3 ] , and Diarizen [ 4 ] . Preliminary experiments included Pyannote Speaker Diarization v3.1; however, the newer community-1 pipeline demonstrates consistent improvements across multiple benchmarks. The framework integrates neural speaker embedding extraction with PLDA-based clustering [ 5 ] , providing robust speaker discrimination under domain variability. According to the official model benchmarks, reported diarization error rates (DER) include 11.7 on ALSHELL-4 and 11.2 on VoxConverse. NeMo Sortformer employs an end-to-end transformer architecture but supports a maximum of four speakers per recording, constraining scalability in unconstrained conversational scenarios [ 3 ] . Diarizen reports performance gains over Pyannote v3.1; however, it remains less extensively validated in open-source benchmarking environments [ 4 ] . Considering benchmark performance, clustering robustness, scalability, and implementation stability, we adopt the Pyannote Community pipeline as the diarization backbone. II-B Model Selection For Automatic Speech Recognition (ASR) For Bangla ASR, we evaluated multiple transformer-based architectures built upon the Whisper and wav2vec2 frameworks. Our baseline system employed bengaliAI/tugstugi_bengaliai-asr_whisper-medium, a Bangla fine-tuned Whisper-medium model, using manual chunking for long-form inference. This model was selected due to its strong performance on Bangla benchmarks and prior empirical validation in recent literature. We further experimented with the WhisperX pipeline, combining: anuragshas/whisper-large-v2-bn, tugstugi_bengaliai-asr_whisper-medium, tanmoyio/wav2vec2-large-xlsr-bengali and ai4bharat/indicwav2vec_v1_bengali. The WhisperX framework enables improved timestamp alignment and segmentation through VAD-based chunking and alignment refinement. However, tugstugi_bengaliai-asr_whisper-medium consistently yielded the lowest word error rate (WER). Accordingly, we adopted this model as our primary ASR backbone [ 6 ] . II-C Preprocessing for Speaker Diarization All audio recordings were preprocessed using DEMUCS for speech enhancement. The model was applied to suppress background noise and isolate speech components prior to diarization. No additional spectral transformations or aggressive filtering were applied in order to preserve speaker-discriminative acoustic characteristics. II-D Preprocessing for ASR We evaluated both speech enhancement and segmentation strategies prior to transcription. Although DEMUCS-based denoising was initially explored, it did not improve performance and was therefore excluded from the final system. For segmentation, we compared Silero VAD with a fine-tuned task-specific segmentation model. The fine-tuned model provided more accurate boundary detection and was adopted in the final ASR pipeline. No additional feature engineering was applied, as the transformer-based model operates directly on waveform-derived log-Mel features. II-E Training for Diarization We trained only the segmentation component of the Pyannote diarization-3.1 model for this task. Several training strategies were explored. Initially, we trained the segmentation model exclusively on the competition dataset, which yielded our best performance. We also experimented with a larger multilingual dataset that combined the competition’s Bengali corpus with an additional Hindi dataset. Our rationale was that speaker diarization is generally considered language-agnostic, and thus, the inclusion of more diverse data might improve generalization. However, this approach resulted in slightly worse performance compared to training solely on the Bengali dataset. The training experiments varied primarily in three dimensions: learning rate, batch size, and regularization strength. Lower learning rates with moderate batch sizes and minimal weight decay produced the most stable convergence and ultimately the best diarization scores. Increasing the learning rate and regularization, while reducing batch size, accelerated training but introduced instability and degraded performance. Extending the number of epochs provided marginal improvements but did not surpass the stability achieved with conservative hyperparameters. Overall, the most effective configuration balanced gradual learning with limited regularization, highlighting the sensitivity of diarization segmentation to training dynamics. In addition to training-time optimization, we performed a structured grid search over clustering and post-processing hyperparameters using the training set as a development set. The search was conducted in three phases: (i) clustering threshold sweep with single-pass inference, (ii) caching raw predictions at the optimal threshold, and (iii) post-processing parameters sweep (e.g., minimum duration and merge gaps) without re-running the full pipeline. This staged strategy enabled efficient hyperparameter tuning while minimizing redundant inference overhead. II-F Training Long Form Transcription For long-form transcription, we attempted to fine-tune the ASR model using an external dataset. This approach, however, yielded extremely poor results, most likely due to inconsistencies and noise within the training data. To address this, we integrated the segmentation model previously trained for diarization into our ASR pipeline. The segmentation model was used to generate coherent audio chunks, which were then transcribed by the ASR system. This strategy provided a more reliable framework for handling extended recordings, as the diarization-informed segmentation improved the alignment and consistency of transcription outputs. Fig. 1: Diarization performance results showing training progression and comparative analysis. III Results III-A Results For Diarization After training for approximately 3–4 epochs, our segmentation model achieved a substantial improvement over the baseline diarization error rate (DER) of 0.40, reducing it to 0.199 on the public evaluation set. Building upon this, we applied a two-pass decoding strategy combined with post-processing refinements, which further lowered the public set DER to 0.192. On the private evaluation set, the same approach yielded a DER of 0.27. These results demonstrate that iterative training and multi-stage inference can significantly enhance diarization performance, particularly on the public benchmark, while still maintaining competitive results on