Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?

Luca Modica Filip Landin Mehrdad Farahani Livia Qian, Gabriel Skantze Richard Johansson · 2026

Read Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models? by Luca Modica Filip Landin Mehrdad Farahani Livia Qian, Gabriel Skantze Richard Johansson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22170.txt.

Description

Luca Modica and co-authors ask whether factual recall mechanisms carry over from text to speech in multimodal language models. They use Causal Mediation Analysis to probe how speech language models encode, store, and retrieve factual associations. Initial results with SpiritLM, which joins discrete speech tokens with text, show gaps between text-to-text and speech-to-text recall. The mechanisms for factual recall thus carry over only partly from text to speech. The findings advance understanding of how internal knowledge mechanisms differ across modalities.

Catalog record

CREATORLuca Modica Filip Landin Mehrdad Farahani Livia Qian, Gabriel Skantze Richard Johansson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24250
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22170

Browse the full library · About the Tower of Babel