Vision-Speech Models: Teaching Speech Models to Converse about Images
Amélie Royer *, Moritz Böhle *, Gabriel de Marmiesse, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez, Patrick Pérez · 2025
Read Vision-Speech Models: Teaching Speech Models to Converse about Images by Amélie Royer *, Moritz Böhle *, Gabriel de Marmiesse, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez, Patrick Pérez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15633.txt.
Introduction
Amélie Royer and six co-authors teach a speech model to talk about images. Their 2025 paper extends a dialogue speech model, Moshi, into a vision-speech system called MoshiVis. Text-based vision models are common, but a voice that can discuss what it sees must keep tone and timing, work in real time, and learn from scarce paired data.
Light adapter modules feed image features into the speech model without retraining its core. A gating switch helps the model move between image talk and plain chat. Training mixes image-text pairs, which are plentiful, with a smaller set of image-speech pairs. The team test the model on visual tasks prompted by voice and by text, and share sample dialogues. Code and the audio test data will be released.
Catalog record
| CREATOR | Amélie Royer *, Moritz Böhle *, Gabriel de Marmiesse, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez, Patrick Pérez |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15633
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision-Speech Models: Teaching Speech Models to Converse about Images Amélie Royer *, Moritz Böhle *, Gabriel de Marmiesse, Laurent Mazaré, Neil Zeghidour, Alexandre Défossez, Patrick Pérez Abstract The recent successes of Vision-Language models raise the question of how to equivalently imbue a pretrained speech model with visual understanding, an important milestone towards building a multimodal speech model able to freely converse about images. Building such a conversational Vision-Speech model brings its unique challenges: (i) paired image-speech datasets are much scarcer than their image-text counterparts, (ii) ensuring real-time latency at inference is crucial thus bringing compute and memory constraints, and (iii) the model should preserve prosodic features ( e.g . , speaker tone) which cannot be inferred from text alone. In this work, we introduce MoshiVis, augmenting a recent dialogue speech LLM, Moshi, with visual inputs through lightweight adaptation modules. An additional dynamic gating mechanism enables the model to more easily switch between the visual inputs and unrelated conversation topics. To reduce training costs, we design a simple one-stage, parameter-efficient fine-tuning pipeline in which we leverage a mixture of image-text ( i.e . , “speechless”) and image-speech samples. We evaluate the model on downstream visual understanding tasks with both audio and text prompts, and report qualitative samples of interactions with MoshiVis. Our inference code, the image-speech data used for audio evaluation, as well as additional information are available at github.com/kyutai-labs/moshivis . † † footnotetext: ∗ Equal contribution. 1 Introduction Figure 1 : MoshiVis is a Vision-Speech model ( VSM ) able to hold full-duplex real-time conversations about an image, and trained with a light data- and compute- budget. For image representations, we use off-the-shelf transformer-based image encoders from the PaliGemma family [ 5 ] . For the speech modelling part, we rely on Moshi [ 8 ] , a recent speech LLM which jointly outputs text and audio tokens in real-time, allowing for full-duplex conversations. At its core, Moshi consists of a standard 7B decoder-only transformer taking as inputs speech tokens (which are the sums of temporally aligned text tokens and audio tokens extracted from the assistant’s and user’s streams), rather than only text like a standard LLM. The output of the transformer is then separately decoded in a text token, as well as passed through a small depth transformer which auto-regressively produces a hierarchy of audio codebooks, then decoded into audio frames. First, ( Sec. 3.1 ), we detail how we augment the speech LLM’s transformer with lightweight visual adaptation modules through cross-attention (CA). We then describe our one-stage finetuning pipeline for these modules: We use a mixture of (i) ( Sec. 3.2 ) image+text only data ( “speechless” data), which, despite incurring a distribution shift due to the lack of audio supervision, allows us to leverage the large body of existing Vision-Language datasets, and (ii) ( Sec. 3.3 ) synthetic spoken visual dialogues which we design to mimic realistic discussions about images. Vision Language Models (VLMs) have recently gained increasing attention, e.g . [ 2 , 9 , 5 , 36 , 4 , 44 , 23 ] , showcasing strong capabilities across a variety of visual understanding tasks such as question answering, image captioning or complex reasoning over visual inputs. A core challenge of training VLMs, or multimodal models in general, is to build well-aligned embeddings of the different input modalities. To this end, the VLM research community has built up vast datasets of paired image and text data over the years, covering many vision understanding tasks [ 11 , 9 , 32 , 21 , 18 , 19 ] . In comparison, such public datasets are very rare in the speech domain, and often limited to captions [ 15 , 6 ] . This lack of data is particularly apparent when considering the challenge of building open-source multimodal models able to naturally talk about an image as well as other general topics, even though such models are starting to appear in the commercial space [ 12 , 31 ] . In this work, we aim to effectively integrate vision capabilities into a conversational speech LLM. As our backbone, we use Moshi [ 8 ] , a recent open-weight speech LLM able to dialogue with the user in real time and in full-duplex, i.e . it is able to listen and speak at any time and does not need to be signalled when to talk. Drawing inspiration from VLMs, we aim to adapt Moshi into a Vision-Speech Model (VSM) with the same dialoguing abilities. We identify three key challenges specific to building a VSM able to hold natural conversations about visual inputs: (i) overcome the above-mentioned scarcity of image-aligned speech data, and avoid blowing up the complexity of the training pipeline as we are now dealing with three modalities—vision, language and audio, (ii) comply with compute and memory constraints in order to hold real-time conversations at inference, and (iii) maintain the original conversational abilities of the backbone dialogue model, i.e . , preserve audio quality as well as prosodic features, and enable seamless switching between image-related and general conversation topics. We address each of these challenges as follows: First, we show that we can adapt the underlying pretrained speech transformer to image inputs using image-text datasets without audio supervision (“speechless” datasets), in combination with a small percentage of speech samples. Specifically, we exploit the fact that Moshi jointly predicts text and audio tokens in a temporally aligned manner. While the produced text tokens differ in distribution from standard language, we find that this form of weak supervision still allows for information transfer from the text to speech modality, despite being out-of-distribution for the backbone model. Second, inspired by recent work on perceptual augmentations [ 34 , 38 ] , we inject the visual inputs into the speech LLM backbone through lightweight adaptation modules based on cross-attention. At inference, the keys and values of the attention mechanism can be efficiently cached and thus only need to be computed once for every image. Third, to ensure that the base model is still able to discuss general conversation topics other than the input image, we design these cross-attention layers to be able to selectively gate visual inputs based on the conversational context. In addition, expanding on previous work on visual dialogues [ 7 , 40 ] , we design a fully synthetic data pipeline to generate short realistic conversations about images, allowing the model to go beyond the usual setting of “one question - one answer” assumed in most VLM benchmarks. An overview of our proposed model, MoshiVis is given in Figure 1 : We augment a pretrained spoken dialogue model, Moshi, with lightweight adaption modules and a simple training pipeline leveraging both image-text and image-speech data. To assess the model’s visual and conversational abilities, we first evaluate MoshiVis’s visual understanding on downstream tasks commonly used in the VLM literature, such as captioning or visual question answering, in both the text and audio realms. To foster further research on Vision-Speech models, we release the audio datasets we use for benchmarking. We then evaluate the model’s ability to switch contexts by measuring how its text reasoning and visual understanding abilities are affected when adding an irrelevant conversation snippet as a prefixing context.