How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue

Hui Lu, Xueyuan Chen, Huimeng Wang, Shuhai Peng, Shiyin Kang, Xixin Wu, Zhiyong Wu · 2026

Read How Should LLMs Listen While Speaking? A Study of User-Stream Routing in Full-Duplex Spoken Dialogue by Hui Lu, Xueyuan Chen, Huimeng Wang, Shuhai Peng, Shiyin Kang, Xixin Wu, Zhiyong Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10199.txt.

Description

Lu and six colleagues study a core design question for full-duplex dialogue, where a model must listen while speaking its own response. Large language models produce one coherent sequence and cannot easily take user input mid-generation. How the user stream enters the model is a key design choice. The authors extend a text-only model into a unified full-duplex system and compare two routing strategies in one pipeline.

Channel fusion injects the user stream directly into the model input, giving stronger semantic grounding and better answering scores. Cross-attention routing instead keeps the user stream as external memory reached through adapters, preserving context better. Under overlapping speech, fusion suffers more context corruption. When the model cannot stop in time, the intruding stream can derail it into incoherent output. Routing, the paper concludes, is a design choice trading integration for robustness.

Catalog record

CREATORHui Lu, Xueyuan Chen, Huimeng Wang, Shuhai Peng, Shiyin Kang, Xixin Wu, Zhiyong Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10199

Browse the full library · About the Tower of Babel