UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction

Yadong Li Guoxin Wu Haiping Hou Biye Li · 2026

Read UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction by Yadong Li Guoxin Wu Haiping Hou Biye Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19221.txt.

Introduction

Li and three colleagues propose the first unified audio front-end model for full-duplex speech, reformulating tasks into a single sequence prediction problem. Cascaded pipelines suffer latency, information loss, and error propagation. End-to-end audio models unify understanding and generation but remain half-duplex, relying on separate voice and turn-taking detectors. Front-end work matters as much as back-end work for seamless interaction. It takes streaming fixed-duration chunks, such as 600 milliseconds, with a reference prompt anchoring the speaker, generating discrete tokens encoding semantic content.

Catalog record

CREATORYadong Li Guoxin Wu Haiping Hou Biye Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43448
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction Yadong Li Guoxin Wu Haiping Hou Biye Li Abstract Full-duplex speech interaction, as the most natural and intuitive mode of human communication, is driving artificial intelligence toward more human-like conversational systems. Traditional cascaded speech processing pipelines suffer from critical limitations, including accumulated latency, information loss, and error propagation across modules. To address these issues, recent efforts focus on the end-to-end audio large language models (LLMs) like GPT-4o, which primarily unify speech understanding and generation task. However, most of these models are inherently half-duplex, and rely on a suite of separate, task-specific front-end components, such as voice activity detection (VAD) and turn-taking detection (TD). In our development of speech assistant, we observed that optimizing the speech front-end is equally crucial as advancing the back-end unified model for achieving seamless, responsive interactions. To bridge this gap, we propose the first unified audio front-end LLM (UAF) tailored for full-duplex speech systems. Our model reformulates diverse audio front-end tasks into a single auto-regressive sequence prediction problem, including VAD, TD, speaker recognition (SR), automatic speech recognition (ASR) and question answer (QA). It takes streaming fixed-duration audio chunk (e.g., 600 ms) as input, leverages a reference audio prompt to anchor the target speaker at the beginning, and regressively generates discrete tokens encoding both semantic content and system-level state controls (e.g., interruption signals). Experiments demonstrate that our model achieves leading performance across multiple audio front-end tasks and significantly enhances response latency and interruption accuracy in real-world interaction scenarios. 1 Introduction Figure 1: Unified Audio Front-end LLM and traditional cascade architecture of full-duplex systems. Speech interaction, as the most natural and efficient form of human communication, is driving artificial intelligence toward more human-like conversational systems. Human conversation is inherently full-duplex: participants speak, listen, and interrupt each other fluidly, relying on subtle acoustic and linguistic cues to maintain coherence and responsiveness, such as voice activity, speaker identity, turn boundaries, and contextual intent. Replicating this natural interaction in artificial systems demands not only intelligent language understanding and generation but also a robust, low-latency audio front-end capable of real-time perception under complex acoustic conditions. As Figure 1 shows, conventional speech interaction systems predominantly rely on complex cascaded pipelines, which are composed of front-end and back-end models. Raw audio first passes through a series of specialized front-end modules, including acoustic echo cancellation (AEC), automatic noise suppression (ANS), voice activity detection (VAD), speaker recognition (SR), and turn-taking detection (TD), and then will be fed into back-end models like Automatic Speech Recognition (ASR), Large Language Model (LLM) for question answer (QA), and Text-to-Speech (TTS). While it is effective to control settings, this pipeline suffers from fundamental limitations in real-world full-duplex scenarios. • First, error propagation and nonlinear distortion is unavoidable in cascaded systems, degrading overall reliability. For example, traditional signal processing front-ends tend to improve the signal-to-noise ratio (SNR) as a single goal, and noise reduction algorithms based on spectral subtraction often destroy the spectral structure of weak speech while suppressing noise, resulting in a significant decline of the ASR Model. • Second, the disjoint optimization of individual tasks prevents the system from leveraging cross-task dependencies and semantic-agnostic decision. For example, traditional VAD models make judgments based only on energy or spectral features, the user’s tone words when thinking or non-interactive voices in the background are easily misjudged as interruption signals, resulting in frequent false interruption. • Third, each module in cascaded architecture introduces computational redundancy and latency accumulation. Consequently, the end-to-end delay is difficult to compress to the comfort zone of human perception (usually 200ms-500ms), making timely interruption (e.g., “stop!” during system playback) difficult to achieve. Recent advances in speech foundation models and end-to-end spoken language systems, exemplified by GPT-4o [ 23 ] , have demonstrated remarkable progress. Many recent end-to-end models unify audio back-end tasks (i.e., speech understanding and generation task) within a single large language model (LLM) framework, where LLMs take speech representations as input and generate both text tokens and speech tokens simultaneously, such as Kimi-Audio [ 19 ] , Step-Audio 2 [ 16 ] , MiMo-Audio [ 29 ] , GLM-4-Voice [ 37 ] , Longcat-Flash-Omni [ 30 ] , Qwen2.5-Omni [ 34 ] and Qwen3-Omni [ 35 ] . These models rely on both large-scale audio-text pre-training and post-training to develop strong audio capabilities. However, most of these models are inherently half-duplex, which rely on pluggable VAD and TD modules to support barge-in [ 4 ] , and they also usually need external front-end processing models to handle real-world challenges like far-field pickup, background noise, acoustic echo, and overlapping speech. Therefore, they still face the drawbacks of the cascading scheme mentioned earlier. In our development of speech assistant, we found that front-end robustness is as critical as back-end intelligence for user satisfaction. Delays in detecting a user’s interruption or misattribution of speaker turns can break the illusion of natural conversation, regardless of how fluent the generated response may be. This observation motivates a paradigm shift: instead of treating front-end tasks as preprocessing steps, we propose to embed them directly into a unified LLM-based generative framework. To this end, we introduce a Unified Audio Front-End (UAF) LLM, the first large language model explicitly designed to unify core audio front-end tasks for full-duplex speech interaction. Our model reformulates VAD, TD, SR, ASR and QA task as a single sequence-to-sequence prediction problem. It operates on streaming audio in fixed-duration chunks (e.g., 600 ms), incorporates a reference audio prompt to lock onto the target speaker, and outputs a discrete token sequence that jointly encodes semantic content (ASR text and model response) and system-level control signals (e.g., “user speaking,” “turn end,” “interruption detected”). By training on diverse real-world interaction data including simulated echo, noise, and overlapping speech, our model learns implicit cross-task representations that outperform cascaded baselines. Experiments show that our UAF achieves state-of-the-art results across individual front-end tasks. More importantly, it enables a truly integrated architecture where perception and action are co-designed within a single generative framework. 2 Related works 2.1 Speech Front-end Processing in Full-duplex Systems As Figure 1 shows, traditional full-duplex speech assistants rely on a cascade of signal processing modules. Acoustic echo cancellation (AEC) is typically handled by adaptive filters or deep learning models such as DCCRN-Echo [ 15 ] . Voice activity detection (VAD) and speaker recognition (SR) are often implemented as separate systems. Early works on VAD relied on hand-crafted acoustic features, such as energy ratio, zero-crossing rate, and signal periodicity [ 24 , 25 , 17 ] . Currently, VAD models based on neural network have gained significant attention in speech research, such as deep neural networks (DNN) [ 18 ] and feedforward sequential memory networks (FSMN) [ 41 , 40 ] . Traditional SR frameworks typically extracts speaker embedding to represent speaker characteristics, and then employ clustering to group speaker embeddings or end-to-end neural diarization for identifying speakers [ 5 ] . Turn-taking detection (TD), though less standardized, usually leverages prosodic cues or dialogue context, such as TEN Turn Detection [ 32 ] , Silero VAD [ 31 ] , and FSMN-VAD [ 40 ] . However, these approaches remain confined to enhancement tasks and do not incorporate higher-level semantic or interaction-aware signals like speaker

Read the complete text