Spoken question answering for visual queries
Nimrod, Zvi, Avihu, Hagai, Ron, Assaf · 2025
Read Spoken question answering for visual queries by Nimrod, Zvi, Avihu, Hagai, Ron, Assaf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23308.txt.
Introduction
The authors build a system for spoken visual question answering in a 2025 paper. Standard question answering uses text. Visual systems add images, and spoken systems add speech. Their goal is a system that takes both speech and images, so a user can ask aloud about a picture.
No data set covered text, speech, and images together, so the team synthesized spoken questions with two text to speech models. A model trained only on synthetic speech came close to one trained on text questions. The choice of speech model had little effect on accuracy, suggesting synthetic speech is a workable path.
Catalog record
| CREATOR | Nimrod, Zvi, Avihu, Hagai, Ron, Assaf |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spoken question answering for visual queries Nimrod, Zvi, Avihu, Hagai, Ron, Assaf Abstract Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively. This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images. Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy. 1 1 1 This is an extended version of the paper that was accepted for Interspeech 2025. This version contains additional tests results. Shabtay Kons Dekel Aronowitz Hoory Arbelle IBM ResearchIsrael Tel-Aviv UniversityIsrael keywords Large language models, Large multimodal models, spoken question answering, visual question answering, multimodal † † email: [email protected], [email protected] 1 Introduction As the AI landscape continues to evolve, there is a growing shift from unimodal to multimodal tasks. Unimodal tasks are those that rely on a single type of input data, such as text, images, or audio, to perform a specific task, such as text generation, question answering, image classification, object detection, and automatic speech recognition (ASR). In contrast multimodal tasks process and integrate multiple types of input data, combining text with images (e.g., VQA [ 1 ] , text-to-image synthesis [ 2 ] ), or combining text with audio (e.g., SQA [ 3 ] , and text-to-speech synthesis [ 4 , 5 ] ). Large language models (LLMs) have become the primary tool for solving many text-based tasks, driven by the abundance of textual training data and the effort put into their training. As these models gain wider adoption, more attention is directed at the human-machine interface. Two crucial modalities for enhancing interaction and usability are speech and vision, i.e. the ability of the models to listen to the user’s speech and to view relevant information in pictures. Speech enables more natural and intuitive communication while vision allows models to process a broader context leading to richer, more interactive and holistic AI systems. In this paper we look at a model that can combine information from the three modalities of text, images and speech. Our focus is on the spoken visual question answering task (SVQA). In this task we give the model an audio with a spoken query. The query is about the visual information within a given image. The query may also include a text prompt with additional information (e.g. instructions and multiple-choice answers). The model is expected to provide a textual response. Several approaches for augmenting an LLM with additional modalities have been explored. Due to the close relation between spoken and written text, one simple approach is to construct a pipeline that first converts the speech to text using an ASR model and then processes the text with an LLM. While this does not require any modifications to the LLM, it is clear that some audible information may be lost in the process and this is highly dependent on the accuracy of the ASR. When dealing with visual information, however, constructing such a pipeline is far from optimal. To that end, models such as LLaVA [ 6 ] propose a modality alignment approach, where an encoded representation of the image, extracted from a pre-trained image encoder, is directly aligned to the LLM’s input embedding space. We extend this approach to a Speech-Vision-Language model by jointly aligning two modality-specific encoders, Whisper [ 7 ] for speech and CLIP [ 8 ] for images, to the LLMs input embedding space. The outline for our model’s architecture is detailed in Section 3 starting from a baseline model that was pre-trained for VQA tasks. To incorporate speech capabilities, we first pre-train the speech modules on speech-only tasks before fine-tuning the entire model on spoken VQA (SVQA) tasks. This process requires suitable training and evaluation datasets, which include both human speech and text-to-speech (TTS)-generated speech. Section 4 provides details on the datasets used and discusses the limitations of using synthesized speech data, along with the methods we employed to mitigate these challenges. In Section 5 , we describe our two-phase training process: (1) separate pre-training of the speech and vision projectors and (2) fine-tuning the joint model. We perform an extensive ablation study by training several models with different combinations of datasets and examine the impact of various training choices. We evaluate our models on several VQA and SVQA test sets, with the results presented in Section 6 . Our model achieves an accuracy of 62% on SEED-Bench, despite the greater complexity of SVQA compared to VQA. This result highlights the potential of extending standard vision-language models to handle more than two modalities. Finally, in Section 7 , we provide a discussion of our results and propose future research directions. We summarize our contributions as follows: 1. We extend VQA models to accept spoken inputs. 2. We create new datasets for spoken visual question answering (SVQA). 3. We demonstrate the practical use of a zero-shot multi-speaker TTS system to synthesize QAs, overcoming previous challenges and preventing training-test speaker leakage. 4. We show that our models perform nearly as well as the textual upper bound, highlighting SVQA’s potential in human-machine interaction. 2 Related work Our work on SVQA extends two important tasks, VQA and SQA. Visual question answering (VQA) attempts to describe, locate, and reason regarding some visual input [ 9 , 10 , 11 ] . Several models that leverage powerful LLMs and vision encoders have been proposed [ 6 , 12 , 13 , 14 ] . Spoken question answering attempts to answer a user’s spoken question [ 15 ] . Several works address this challenge [ 3 , 16 ] . Spoken visual question answering was tackled by several works, which leveraged a TTS system to synthesize the questions [ 17 , 18 , 19 ] . These works use a single-speaker TTS system. This may not generalize to unseen speakers, as their models have only seen a single speaker during training. Moreover, they test on data synthesized from the same TTS system, which does not expose this limitation. Furthermore, they formulate the problem as a classification rather than a generative task. A recent work [ 20 ] uses cross-attention of audio/vision embeddings, resembling contrastive loss, making it more of a classifier than a generative model. Other works focus on SQA for the medical domain [ 21 , 22 ] . Another related area of works that incorporate text, audio, and images is audio-visual question answering, which is a part of a more general task of video understanding [ 23 , 24 , 25 , 26 ] . 3 Spoken VQA model architecture Our model is based