Closing the Modality Reasoning Gap for Speech Large Language Models
Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li, Zhizheng Wu · 2026
Read Closing the Modality Reasoning Gap for Speech Large Language Models by Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li, Zhizheng Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05543.txt.
Introduction
Chaoren Wang and 6 co-authors tackle, in a 2026 paper, the gap between how speech language models reason from speech and from text. Given the same content as text, these models usually reason better. The authors link that gap to hidden representations drifting apart across layers and to behavior wandering during long reasoning chains.
Their framework, TARS, trains models with reinforcement learning and two alignment rewards. One reward measures how similar hidden states stay between speech-based and text-based runs. The other checks whether outputs stay semantically consistent with reference text completions. On the MMSU and OBQA benchmarks, the method narrowed the gap and reached leading results among speech models of its size class.
Catalog record
| CREATOR | Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li, Zhizheng Wu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05543
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Closing the Modality Reasoning Gap for Speech Large Language Models Chaoren Wang, Heng Lu, Xueyao Zhang, Shujie Liu, Yan Lu, Jinyu Li, Zhizheng Wu Abstract Although speech large language models have achieved notable progress, a substantial modality reasoning gap remains: their reasoning performance on speech inputs is markedly weaker than on text. This gap could be associated with representational drift across Transformer layers and behavior deviations in long-chain reasoning. To address this issue, we introduce TARS , a reinforcement - learning framework that aligns text - conditioned and speech - conditioned trajectories through an asymmetric reward design. The framework employs two dense and complementary signals: representation alignment, which measures layer-wise hidden-state similarity between speech- and text-conditioned trajectories, and behavior alignment, which evaluates semantic consistency between generated outputs and reference text completions. Experiments on challenging reasoning benchmarks, including MMSU and OBQA, show that our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs. 1 Introduction Recent advances in Speech Large Language Models (Speech LLMs) enable a unified framework for spoken language processing tasks such as automatic speech recognition (ASR), speech translation, and speech QA. Most Speech LLMs follow a three-stage architecture, consisting of a pre-trained speech encoder, lightweight adapters, and a decoder-only text LLM Peng et al. 2025 ; Cui et al. 2025 . The encoder transforms raw speech into high - resolution acoustic representations, which are then projected into the text - embedding space through learned adapters, enabling the downstream LLM to process speech inputs using its text - native reasoning stack. Through shared representations across modalities, this architecture allows speech inputs to leverage the generation and reasoning capabilities of text-based LLMs. However, Speech LLMs exhibit a persistent and critical challenge: the modality reasoning gap, denoting a substantial decline in reasoning performance on speech inputs compared to text, as evidenced by empirical analyses Xiang et al. 2025 on VoiceBench and SpeechMMLU benchmarks Chen et al. 2024 ; Xiaomi 2025 . To close modality reasoning gap, prior works have primarily focused on input-side fusion and output-side supervision. Input-level modality fusion methods aim to reduce the discrepancy between speech representations and text embeddings at input stage, by freezing the LLM backbone and training lightweight adapters Anonymous 2025 ; Lu et al. 2025a ; Xu et al. 2025c . As speech naturally contains paralinguistic cues absent in text, strict input equivalence may not be desirable. However, for complex reasoning tasks, the underlying logical progression should remain invariant regardless of the input modality Mousavi et al. 2025 . Relying solely on inputs can cause subtle discrepancies to propagate and amplify through Transformer layers, leading to significant representational drift. Another line of work provides output-level supervision. They focus on knowledge distillation or prompt-switching training to encourage speech-conditioned generations to mimic text-conditioned behaviors Wang et al. 2025 ; Ding et al. 2025 . However, these methods enforce strict token-level supervision in a off-policy manner. Since the speech-conditioned distribution differs from the text one, forcing the model to generate exact text-conditioned tokens targets an unreachable objective. Furthermore, this rigid supervision suffers from exposure bias: a single token error shifts the model into an unsupervised state, causing it to fall into behavioral divergence. To address these limitations, we introduce TARS ( T rajectory A lignment for R easoning in S peech), which combines representation alignment to mitigate drift and behavior alignment with more flexible objective. By leveraging on-policy exploration, our method enables the model to mitigate exposure bias and maintain reasoning stability, preventing the model from drifting into out-of-distribution states. Specifically, representation alignment is computed from the cosine similarity of layer - wise hidden states, providing coarse - grained representation - level feedback. Complementarily, behavior alignment is derived from the semantic consistency of the final outputs, providing token - level but comparatively sparser feedback. These signals effectively steer the speech modality toward the text reasoning trajectory. Under an asymmetric reward design with Group Relative Policy Optimization (GRPO) Shao et al. 2024 , we jointly optimize task accuracy and these two alignment rewards. Even when all samples in the generated group exhibit zero task accuracy, a common outcome for speech - conditioned reasoning given its greater difficulty than text - based inference, the alignment signals remain informative, enabling direct trajectory alignment between speech and text. Empirical results on complex reasoning benchmarks, such as MMSU and OBQA, demonstrate that our method outperforms existing baselines, achieving state-of-the-art performance among 7B models. The contributions of our work are summarized as follows: • We propose an on-policy reinforcement-learning framework for trajectory alignment that aligns speech-conditioned reasoning trajectory with its text-conditioned counterpart, closing the modality reasoning gap without architectural modifications. • We introduce an asymmetric dense reward with two complementary alignment signals: representation alignment that reduces layer-wise hidden-state drift, and behavior alignment that enforces semantic consistency. • Our method achieves the state - of - the - art performance on reasoning benchmarks (MMSU, OBQA) among 7B - scale models. We release datasets and code to facilitate reproducibility. 2 Related Works Speech LLMs. Speech LLMs have progressed from cascaded pipelines (ASR + text LLM + optional TTS) to end-to-end architectures that couple speech perception with LLM-style generation, enabling spoken dialogue and spoken QA while better leveraging paralinguistic cues beyond transcripts Peng et al. 2025 ; Cui et al. 2025 . A dominant design follows a three-stage paradigm: a pretrained speech encoder extracts acoustic features, which are mapped into the text embedding space via lightweight projectors to condition a decoder-only LLM, preserving text-pretrained reasoning priors while extending to speech tasks. Recent open and proprietary omni systems further target low-latency and multi-modal interaction. Open-weight examples such as Qwen2.5-Omni and Qwen3-Omni integrate unified perception and generation, and introduce modality-specialized capacity (e.g., MoE routing) to improve scalability Xu et al. 2025a ; Xu et al. 2025b . Audio-centric models like Kimi-Audio and MiniCPM-o emphasize practical voice interaction and general audio understanding Ding et al. 2025 ; Yao et al. 2024 . Modality Alignment and Reasoning Gap. Despite the unified architecture, a performance disparity between speech and text modalities persists, termed the modality reasoning gap. Empirical studies Chen et al. 2024 ; Xiaomi 2025 ; Mousavi et al. 2025 and representational analyses Xiang et al. 2025 reveal that speech-conditioned hidden states often drift from their text counterparts, leading to degraded reasoning capabilities. Existing efforts to bridge this gap generally fall into two categories: (1) Architectural Adaptations. To prevent the degradation of text-based capabilities, a prominent line of work adopts a frozen-backbone strategy. Methods such as AlignChat Anonymous 2025 , DeSTA Lu et al. 2025a , OTReg Xu et al. 2025c , and MTBI Xie et al. 2025 keep the LLM parameters fixed, focusing exclusively on refining the input-side projector. For instance, DeSTA utilizes speech captioning tasks to force acoustic features into the text embedding space. However, this approach yields only a surface-level alignment. By freezing the backbone, the model cannot adapt to speech-specific dynamics, causing reasoning trajectories to diverge even when inputs are projected closely. (2) Supervised Alignment Strategies. Beyond architectural changes, other works employ data-driven objectives to align model behaviors at the output level. Approaches such as Kimi-Audio Ding et al. 2025 utilize prompt-switching strategies to bridge generation divergence, while others apply cross-modal