HITSZ’s End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track

Xuchen Wei , Yangxin Wu , Yaoyin Zhang , Henglyu Liu, Kehai Chen, Xuefeng Bai, Min Zhang School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China {2023311524,2023311526,2023313720,23s151043}@stu.hit.edu.cn, {chenkehai,baixuefeng,zhangmin2021}@hit.edu.cn · 2025

Read HITSZ’s End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track by Xuchen Wei , Yangxin Wu , Yaoyin Zhang , Henglyu Liu, Kehai Chen, Xuefeng Bai, Min Zhang School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China {2023311524,2023311526,2023313720,23s151043}@stu.hit.edu.cn, {chenkehai,baixuefeng,zhangmin2021}@hit.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19616.txt.

Introduction

Xuchen Wei and six co-authors describe a speech translation system for a 2025 shared task on Indic languages. Translating speech between English and the languages of India is a low resource challenge. Their end to end system pairs a strong pretrained speech recognition model with a large language model specialized for Indic languages.

The combined system scored well in both directions on the standard BLEU measure. The authors also tested a chain of thought method that asks the model to reason in steps. On outputs that followed the required format, that method gave large gains for one language pair. Keeping the model inside the format proved inconsistent, which the authors flag as the main hurdle.

Catalog record

CREATORXuchen Wei , Yangxin Wu , Yaoyin Zhang , Henglyu Liu, Kehai Chen, Xuefeng Bai, Min Zhang School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China {2023311524,2023311526,2023313720,23s151043}@stu.hit.edu.cn, {chenkehai,baixuefeng,zhangmin2021}@hit.edu.cn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-19616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HITSZ’s End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Track Xuchen Wei , Yangxin Wu , Yaoyin Zhang , Henglyu Liu, Kehai Chen, Xuefeng Bai, Min Zhang School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China {2023311524,2023311526,2023313720,23s151043}@stu.hit.edu.cn, {chenkehai,baixuefeng,zhangmin2021}@hit.edu.cn Abstract This paper presents HITSZ’s submission for the IWSLT 2025 Indic track, focusing on speech-to-text translation (ST) for English-to-Indic and Indic-to-English language pairs. To enhance translation quality in this low-resource scenario, we propose an end-to-end system integrating the pre-trained Whisper automated speech recognition (ASR) model with Krutrim, an Indic-specialized large language model (LLM). Experimental results demonstrate that our end-to-end system achieved average BLEU scores of 28.88 28.88 for English-to-Indic directions and 27.86 27.86 for Indic-to-English directions. Furthermore, we investigated the Chain-of-Thought (CoT) method. While this method showed potential for significant translation quality improvements on successfully parsed outputs (e.g. a 13.84 13.84 BLEU increase for Tamil-to-English), we observed challenges in ensuring the model consistently adheres to the required CoT output format. † † * Corresponding author 1 Introduction Speech-to-text translation plays a vital role in overcoming language barriers in multilingual and international contexts, such as real-time translation during online meetings. Although translation systems for high-resource language pairs have achieved impressive performance, low-resource language pairs, particularly those involving Indic languages, continue to face significant challenges ( Radford et al. 2023 ; Joshi et al. 2020 ) . This paper presents HITSZ’s submission to the Indic Track of IWSLT 2025, covering bidirectional speech translation between English and three major Indic languages: Hindi, Bengali, and Tamil. An overview of the end-to-end system is illustrated in Figure 1 . Data scarcity poses a significant challenge for speech translation (ST) between English and Indic languages, primarily due to the low-resource nature of these language pairs and the reliance on data-driven neural models ( Ahmad et al. 2024 ) . Acknowledging this, we collected available parallel corpus from the official IWSLT data releases for effective end-to-end ST model training. Cascade and end-to-end (E2E) systems represent two prominent paradigms in ST, each offering distinct advantages ( Ney 1999 ; Mathias and Byrne 2006 ; Bérard et al. 2016 ) . While cascaded systems typically achieve higher translation quality ( Agarwal et al. 2023 ) , E2E systems are favored for their lower latency and reduced modeling complexity ( Ahmad et al. 2024 ; Xu et al. 2023 ) . This work focuses exclusively on the end-to-end paradigm for the bidirectional speech translation task. We adopt an unconstrained setting and utilize state-of-the-art pre-trained models, including Whisper ( Radford et al. 2023 ) and Krutrim ( Kallappa et al. 2025 ) , to develop E2E systems for both English-to-Indic and Indic-to-English directions. Although additional resources such as the IndicVoices ( Javed et al. 2024 ) dataset are available, we deliberately exclude them due to concerns about potential overlap with the test set. The remainder of this paper is structured as follows: Section 2 reviews related work on speech translation, particularly in low-resource and Indic language settings. Section 3 describes the datasets and data pre-processing. Section 4 introduces our end-to-end system. Section 5 presents the experimental settings, results, and analysis. Lastly, Section 6 concludes the paper. 2 Related Work Recent advances in end-to-end speech translation (ST) have demonstrated the effectiveness of combining large pre-trained models with task-specific adaptation ( Wang et al. 2017 ; Bérard et al. 2018 ; Bansal et al. 2019 ; Wang et al. 2020 ; Alinejad and Sarkar 2020 ) , especially in low-resource and multilingual settings ( Marie et al. 2019 ; Sun et al. 2020 ; Tsiamas et al. 2024 ; Li et al. 2025 ) . Among them, several works stand out for their innovative training paradigms and architectural choices that have directly influenced our approach. These include NICT’s submission to IWSLT 2024 ( Dabre and Song 2024 ) , which leverages decoder-side fine-tuning of Whisper with pseudo-labels from IndicTrans2 ( Gala et al. 2023 ) ; ZeroSwot, which introduces an encoder-centric alignment method for zero-shot ST ( Tsiamas et al. 2024 ) ; and SALMONN, a multimodal framework that uses a lightweight training pipeline to adapt frozen encoders and LLMs through cross-modal instruction tuning ( Tang et al. 2024 ) . In what follows, we briefly review each of these works and highlight their relevance to our system design. 2.1 NICT’s E2E ST System in IWSLT 2024 One of the most relevant works to our approach is the IWSLT 2024 submission by NICT, which developed end-to-end speech translation systems for English to Hindi, Bengali, and Tamil. A key contribution was their fine-tuning strategy for Whisper: instead of using human-annotated translations, they first fine-tuned IndicTrans2 to generate pseudo-translations from English transcripts. These synthetic targets were then used to train Whisper in a speech-to-text translation setting, effectively distilling knowledge and improving decoder performance beyond what reference translations alone could achieve. 2.2 ZeroSwot Another influential work is ZeroSwot, which proposes a novel zero-shot end-to-end ST framework by aligning speech representations with the embedding space of a multilingual MT model. In their setup, the speech encoder is initialized from a CTC-finetuned wav2vec 2.0 model Baevski et al. 2020 and trained using a combination of CTC loss and Optimal Transport loss ( Graves et al. 2006 ; Peyré et al. 2019 ) . The goal is to produce subword-level acoustic representations that match those expected by a frozen multilingual MT encoder (NLLB) ( Team et al. 2022 ) . In addition, a compression adapter ( Liu et al. 2020 ) is introduced to map variable-length audio sequences into subword-aligned embeddings, bridging both length and representation mismatches between modalities. In contrast to NICT’s decoder-focused fine-tuning, ZeroSwot emphasizes encoder-side alignment, enabling zero-shot translation without requiring any parallel ST data. 2.3 SALMONN We also take inspiration from SALMONN, a multimodal framework that integrates Whisper and BEATs ( Chen et al. 2023 ) encoders with a large language model (Vicuna) ( Chiang et al. 2023 ) to enable general auditory understanding across speech, audio events, and music. Although SALMONN targets a broader set of audio-language tasks beyond ST, its modular design and training strategy are particularly relevant. SALMONN adopts a three-stage training pipeline—pre-training ( Zhu et al. 2024 ) , instruction tuning, and activation tuning—where only lightweight modules (Q-Former and LoRA adaptors ( Li et al. 2023 ; Hu et al. 2022 ) ) are updated while the encoders and LLM remain frozen. This design enables efficient adaptation with minimal parameter updates. Our work builds on this principle by leveraging pre-trained components and applying modular fine-tuning in a similarly efficient manner, tailored to low-resource, bidirectional speech translation between English and Indic languages. 3 Data In this section, we present the statistics of the initial corpora and describe our methods for pre-processing the raw data. 3.1 Dataset Direction Train Dev Test Total Speech Hours en \rightarrow bn 680.9 40.8 93.2 814.9 en \rightarrow hi 680.9 40.8 93.2 814.9 en \rightarrow ta 680.9 40.8 93.2 814.9 bn \rightarrow en 158.0 1.0 1.3 160.3 hi \rightarrow en 653.9 1.0 1.3 656.2 ta \rightarrow en 478.2 1.0 2.2 481.4 Table 1: Statistics of dataset for training, development, and test sets. The abbreviations en

Read the complete text