Audio–Image Alignment as a Continued-Pretraining Stage Improves Low-Resource ASR
1 st Sujith Pulikodan, 2 nd Nihar Desai, 3 rd Prasanta Kumar Ghosh · 2026
Read Audio–Image Alignment as a Continued-Pretraining Stage Improves Low-Resource ASR by 1 st Sujith Pulikodan, 2 nd Nihar Desai, 3 rd Prasanta Kumar Ghosh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24080.txt.
Introduction
Pulikodan, Desai, and Ghosh look for a cheaper way to adapt speech recognition to languages with little transcribed data. Writing out speech by hand is slow and costly, which leaves thousands of languages poorly served by automatic speech recognition. Their 2026 idea inserts an extra training stage between general pretraining and final fine-tuning, one that needs no transcripts at all and uses pictures instead.
Speech clips from the Vaani dataset were collected using images as prompts, so each recording naturally pairs with a picture. The team aligns sound representations from a pretrained FastConformer audio encoder with image representations from vision encoders, nudging the audio model toward the paired meanings. After this alignment stage, fine-tuned models consistently recognise speech better than models fine-tuned directly. The result suggests audio-image alignment is a practical, transcript-free adaptation step for low-resource languages.
Catalog record
| CREATOR | 1 st Sujith Pulikodan, 2 nd Nihar Desai, 3 rd Prasanta Kumar Ghosh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24080
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio–Image Alignment as a Continued-Pretraining Stage Improves Low-Resource ASR 1 st Sujith Pulikodan, 2 nd Nihar Desai, 3 rd Prasanta Kumar Ghosh Abstract Thousands of languages are spoken worldwide, yet many remain under-resourced for Automatic Speech Recognition (ASR) due to the limited availability of high-quality transcribed speech data. Collecting accurate transcriptions is often costly and labor-intensive, particularly for low-resource languages. In this work, we investigate the use of aligned audio–image pairs to adapt pretrained audio encoders without requiring transcription data before supervised finetuning. Our proposed representation alignment stage is introduced between large-scale pretraining and supervised ASR fine-tuning. Specifically, image representations extracted from pretrained vision encoders are aligned with audio representations to further adapt a pretrained audio encoder. For this alignment process, we utilize the Vaani dataset, in which images serve as prompts for speech collection, naturally providing paired audio–image data. We evaluate the proposed approach using multiple vision encoders and a pretrained FastConformer audio encoder. Experimental results demonstrate that models fine-tuned after representation alignment consistently achieve improved ASR performance compared to direct fine-tuning. These findings highlight the potential of audio–image representation alignment as an effective transcription-free adaptation strategy for enhancing ASR systems in low-resource language settings. I Introduction Pretraining and fine-tuning have become the dominant paradigm for developing AI models, including Automatic Speech Recognition (ASR) systems. In this framework, audio encoders are first pretrained in a self-supervised manner on large-scale unlabeled speech corpora and subsequently fine-tuned using labeled audio–text pairs [ 1 , 2 ] . This approach effectively exploits the abundance of unlabeled speech data while reducing reliance on costly manual annotations. However, existing ASR training pipelines primarily utilize audio-only data during pretraining and audio–text pairs during fine-tuning, leaving other readily available multimodal sources largely unexplored. Many languages around the world have only limited amounts of transcribed speech data available for training Automatic Speech Recognition (ASR) systems. Even within the widely adopted pretraining–fine-tuning paradigm, increasing the amount of labeled data during fine-tuning generally leads to improved performance [ 2 ] . However, obtaining high-quality transcriptions is often expensive, time-consuming, and requires language expertise, making it particularly challenging for low-resource languages. As a result, many languages remain underrepresented, leading to lower ASR accuracy. In contrast, multimodal data consisting of semantically related audio and images can often be collected more easily and at a larger scale than fully transcribed speech. Motivated by this observation, we investigate the use of aligned audio–image pairs to further adapt pretrained audio encoders without requiring transcription labels. Specifically, we introduce an intermediate representation alignment stage between audio pretraining and supervised fine-tuning, where audio representations are aligned with semantic representations extracted from images. By leveraging the complementary semantic information available in images, this adaptation stage aims to improve the quality of speech representations before ASR fine-tuning. We hypothesize that such multimodal adaptation can serve as an effective transcription-free learning signal and improve downstream ASR performance, particularly in low-resource language settings where labeled speech data is scarce. II Related Work Audio-Visual Speech Recognition (AVSR) has been extensively studied as a multimodal approach to speech recognition [ 3 ] . In AVSR systems, audio signals are combined with visual information, such as lip movements or video frames, to improve recognition accuracy, particularly in noisy environments. The audio and visual modalities are jointly processed, and their representations are fused to generate the final transcription. Related approaches have also explored scene-aware speech recognition, where contextual visual information from the surrounding environment is incorporated alongside the audio signal to improve recognition performance [ 4 ] . In most of these works, the primary objective is to combine multiple modalities during inference to enhance ASR robustness and accuracy. In parallel, several self-supervised learning (SSL) approaches have been proposed for learning speech representations from large-scale unlabeled audio data, including wav2vec 2.0 [ 1 ] , HuBERT [ 2 ] , and Best-RQ [ 5 ] . Similarly, vision–language models such as CLIP [ 8 ] and SigLIP [ 10 ] have demonstrated the effectiveness of aligning representations across modalities through contrastive learning. Building on these advances, multimodal models such as SpeechCLIP [ 12 ] and AudioCLIP [ 13 ] learn shared embedding spaces between audio and visual representations for cross-modal retrieval, representation learning, and multimodal understanding tasks. In contrast, our work does not aim to perform multimodal inference or learn a unified multimodal representation space. Instead, we leverage aligned audio–image pairs as a transcription-free adaptation signal to further refine a pretrained audio encoder. Specifically, we introduce an intermediate representation alignment stage between self-supervised audio pretraining and supervised ASR fine-tuning. During this stage, the audio encoder is adapted by aligning its representations with semantic representations extracted from images, before any ASR fine-tuning using transcribed speech data. We investigate whether this additional multimodal adaptation step can improve the quality of speech representations and subsequently enhance downstream ASR performance, particularly in low-resource language settings. III Approach The end-to-end ASR model pipeline typically consists of two stages: pretraining and fine-tuning. In this work, we introduce an additional stage, termed audio–image alignment , between these two stages. During this stage, the parameters of the audio encoder are further optimized using aligned audio–image representations. The underlying assumption is the availability of multimodal data in which semantic correspondence exists between the audio and the associated images. Such data may be easier and less expensive to obtain than manually transcribed speech, as it does not require the creation of text annotations. 1 Pretraining FastConformer audio encoder AUDIO 2 Audio–Image Alignment Contrastive align. to vision enc. AUDIO IMAGE 3 Fine-tuning (ASR) FastConformer + hybrid CTC–TDT decoder AUDIO TRANSCRIPT init. weights init. weights Fig. 1: Three-stage pipeline: audio pretraining \rightarrow audio–image alignment \rightarrow ASR fine-tuning, with weights carried forward between stages. For audio–image alignment, we use a pretrained image encoder to extract image representations. The audio encoder is then trained to align its representations with the image embeddings, while all layers of the audio encoder are updated during this process. To enable this alignment, we introduce additional layers, referred to as the alignment head , which project the audio representations into a space that is compatible with the image embeddings. In the final stage, the aligned audio encoder is combined with a hybrid CTC-TDT decoder and undergoes supervised training using transcribed speech data in a multilingual setting. SigLIP SigLIP-MT Qwen-MT single-token D{=}768 \cdot vision \sim 95M multi-token D{=}1024 \cdot vision \sim 650M multi-token D{=}2048 \cdot vision \sim 500M AUDIO BRANCH IMAGE BRANCH SCORE & LOSS input audio waveform audio waveform audio waveform preprocessor log-mel (frozen) log-mel (frozen) log-mel (frozen) FastConformer 17 blocks all 17 trainable all 17 trainable all 17 trainable audio pool ( K_{a} queries) single-query AttentionPool Multi-query AttnPool, K_{a}{=}16 Multi-query AttnPool, K_{a}{=}16 projection + L2 MLP 1024\!\to\!768 \mathbf{a}\in\mathbb{R}^{768} MLP 1024\!\to\!1024 \mathbf{q}\in\mathbb{R}^{16\times 1024} MLP 1024\!\to\!2048 \mathbf{q}\in\mathbb{R}^{16\times 2048} input image image image vision encoder (frozen) SigLIP2-base patch16-256 SigLIP2-large patch16-384 Qwen3-VL visual (LM stack skipped) token selection pool + proj head \to 1 pooled vector 576 patch tokens \to top-16 by \|\cdot\|_{2} 2{\times}2 spatial merge \to up to 16 tokens cache ( K_{v} tokens) \mathbf{v}\in\mathbb{R}^{768} \mathbf{V}\in\mathbb{R}^{16\times 1024} \mathbf{V}\in\mathbb{R}^{16\times 2048} pair score cosine \mathbf{a}^{\!\top}\mathbf{v} MaxSim \tfrac{1}{K_{a}}\sum_{a}\max_{v}\langle q_{a},V_{v}\rangle MaxSim \tfrac{1}{K_{a}}\sum_{a}\max_{v}\langle q_{a},V_{v}\rangle loss sigmoid SigLIP -\log\sigma(y_{ij}(t\,s_{ij}+b)) sigmoid SigLIP -\log\sigma(y_{ij}(t\,s_{ij}+b)) sigmoid SigLIP -\log\sigma(y_{ij}(t\,s_{ij}+b)) Legend: frozen trainable on-disk cache param-free