SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation

Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang · 2025

Read SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation by Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17060.txt.

Introduction

Yu and co-authors present a 2025 paper on a single speech model that talks and listens at the same time. Full-duplex chat, where both sides can interrupt, usually chains many parts that pass errors along. A recent rival folds audio codes into one model's token stream, but speech quality drops. Their model, SALMONN-omni, stands alone with no codec in its token space.

A dynamic thinking trick inside the model times its switches between speaking and listening. That helps it handle echoes and context-based interruptions. On spoken question and open chat benchmarks, it beat open full-duplex rivals by at least thirty percent in relative terms. It also neared half-duplex systems that never overlap speech.

Catalog record

CREATORWenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang Abstract In order to enable fluid and natural human-machine speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such as voice activity detectors, interrupters, conversation state predictors, or multiple LLMs. These systems, however, suffer from error accumulation across modules and struggle with key challenges such as context-dependent barge-in and echo cancellation. Recent approaches, most notably Moshi, simplify the pipeline by injecting audio codecs into the token space of a single LLM. However, such methods still incur significant performance degradation when operating on the speech rather than text modality. In this paper, we introduce SALMONN-omni, the first single, standalone full-duplex speech LLM that operates without audio codecs in its token space. It features a novel dynamic thinking mechanism within the LLM backbone, enabling the model to learn when to transition between speaking and listening states. Experiments on widely used benchmarks for spoken question answering and open-domain dialogue show that SALMONN-omni achieves at least 30% relative performance improvement over existing open-source full-duplex models and performs highly competitively to half-duplex and turn-based systems, despite using substantially less training data. Moreover, SALMONN-omni demonstrates strong performance in complex conversational scenarios, including turn-taking, backchanneling, echo cancellation and context-dependent barge-in, with further improvements achieved through reinforcement learning. Some demo conversations between user and SALMONN-omni are provided in the following repository https://github.com/bytedance/SALMONN . 1 Introduction Large language models (LLMs) [ 1 , 2 , 3 ] have revolutionized problem-solving and task execution through text-based interaction. However, speech, being a more natural and intuitive modality, offers a smoother and more human-centric interface for human-machine communication. As a result, there is increasing interest in extending text-based LLMs with speech-processing capabilities. These efforts span a range of tasks, including advanced speech and audio understanding [ 4 , 5 , 6 ] , text-driven speech synthesis [ 7 , 8 , 9 ] and interactive spoken dialogue systems [ 10 , 11 , 12 ] . While turn-based speech LLMs [ 10 , 12 , 13 , 14 , 15 ] support conversational artificial intelligence (AI) in a half-duplex manner, human conversation is inherently full-duplex, requiring the ability to listen, think, and speak simultaneously. This dynamic is evident in natural conversational behaviours such as frequent turn-taking, backchanneling, overlapping speech and barge-ins, all of which contribute to more fluid and engaging interactions. These rich, interactive communication patterns have spurred growing interest in full-duplex speech LLMs, which seek to improve the naturalness, responsiveness and overall quality of human-AI conversations. The recent release of GPT-4o (GPT-4omni) has demonstrated low-latency, emotionally expressive speech capabilities, further increasing global interest in full-duplex speech technologies. Among open research efforts, models such as Moshi [ 11 ] , SyncLLM [ 16 ] and OmniFlatten [ 17 ] inject discrete audio codec tokens into pre-trained LLM backbones and fine-tune them to support flexible speech input and output. However, these approaches typically require large-scale training data to bridge the modality gap between text and speech and to mitigate catastrophic forgetting of original LLM textual knowledge. As a result, the speech synthesizer must infer whether to produce speech or silence based solely on the codec stream. While these models avoid the need for annotations of conversational dynamics, they frequently struggle with accurate timing of dialogue transitions [ 18 ] . Other approaches [ 19 , 20 , 21 , 22 ] explicitly predict transitions between speaking and non-speaking states and integrate streaming speech encoders and synthesizers to enable real-time interaction. However, these systems typically rely on complex mechanisms rather than a standalone LLM to handle full-duplex functionality, resulting in added system complexity and suboptimal context integration. Some methods, such as [ 21 , 22 ] , propose cascaded systems with external modules like voice activity detection (VAD) and dialogue controllers to manage conversational flow, which suffers from error propagation. Others, including VITA [ 19 ] and Freeze-Omni [ 20 ] , follow a “model-as-a-server” paradigm, running two interdependent LLM processes, with one dedicated to listening and the other to speaking, thus introducing additional computational and memory overhead. This paper introduces SALMONN-omni, a standalone speech interaction model based on a novel codec-free, full-duplex spoken dialogue framework. By seamlessly integrating a streaming speech encoder, a single LLM backbone without audio codec injection, and a streaming speech synthesizer, SALMONN-omni supports simultaneous speech input and output within a unified end-to-end architecture. To autonomously manage dialogue state transitions in full-duplex interactions, we propose a periodic synchronisation mechanism coupled with a novel “thinking”’ strategy, endowing the model with the temporal awareness that enables coherent and responsive real-time speech communication. Our key contributions are summarised as follows: • We present SALMONN-omni, the first standalone speech LLM that enables full-duplex human-AI conversations without injecting audio codecs into the LLM backbone. Unlike previous models, SALMONN-omni directly integrates a streaming speech encoder, an LLM, and a speech synthesizer, requiring careful synchronisation to process input and output speech simultaneously. As a standalone model, the model must also decide when to start and stop speaking on its own. To achieve this, we train the LLM to generate special tokens that control dialogue timing, treating these state transitions just like generating normal text tokens, based on the idea that “your LLM is secretly a full-duplex predictor.” • Experiments on widely used spoken question answering (QA) and conversation benchmarks demonstrate the strong performance of SALMONN-omni. In full-duplex mode, SALMONN-omni sets a new state-of-the-art (SOTA), achieving at least 30% relative improvement over previous best results. In half-duplex evaluation, it delivers highly competitive performance compared to recent turn-based models, including those trained on substantially larger datasets – some with up to 13 million hours of audio [ 21 ] . • SALMONN-omni shows strong performance in predicting the timing of turn-taking, backchanneling, and context-dependent barge-ins in free-form conversations. Additionally, to the best of our knowledge, we are the first to incorporate reinforcement learning (RL) to further improve the modeling of dialogue dynamics. 2 Related Work 2.1 End-to-end Speech (Interaction) LLMs Compared with traditional modular conversational AI systems [ 23 , 24 , 25 ] , end-to-end speech interaction LLMs have attracted great attention for their ability to support fluent, expressive, and emotionally rich spoken interactions. Depending on whether the model can listen and speak simultaneously, a core characteristic of human communication, recent end-to-end conversational AI systems can be broadly categorised into two types: half-duplex (turn-based) and full-duplex speech LLMs. Currently, most speech LLMs operate in a half-duplex manner [ 13 , 14 , 19 , 26 , 27 ] , including models such as GLM-4-Voice [ 15 ] , Qwen2.5-Omni [ 22 ] , and Kimi-Audio [ 21 ] . While these models can engage in turn-based speech conversations, they lack an internal duplex strategy for modelling dialogue dynamics like turn-taking. Instead, they rely on external VAD modules to alternate between listening and speaking states. As a result, they struggle with key aspects of natural conversation ( e.g. , barge-ins and backchanneling) which require the ability to listen and speak simultaneously. Full-duplex speech LLMs [ 11 , 16 , 17 , 20 , 28

Read the complete text