Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems

Haoyuan Yu, Yuxuan Chen, Minjie Cai · 2026

Read Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems by Haoyuan Yu, Yuxuan Chen, Minjie Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20230.txt.

Introduction

Haoyuan Yu, Yuxuan Chen, and Minjie Cai present a framework for full-duplex voice dialogue, where a system listens and speaks at the same time. Natural conversation needs this overlap; turn-taking systems feel slow by contrast. The authors break dialogue into minimal conversational units, so each unit can be processed on its own.

Their system predicts when to move from one unit to the next. It is built around a multimodal large language model, with voice activity detection and speech synthesis as supporting parts. The setup works without task training, in a plug-and-play manner. Smoother voice dialogue would help assistants feel more like talking with a person.

Catalog record

CREATORHaoyuan Yu, Yuxuan Chen, Minjie Cai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE8 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20230

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems
Haoyuan Yu, Yuxuan Chen, Minjie Cai

Abstract
Full-duplex voice interaction is crucial for natural human computer interaction. We present a framework that decomposes complex dialogue into minimal conversational units, enabling the system to process each unit independently and predict when to transit to the next. This framework is instantiated as a semi-cascaded full-duplex dialogue system built around a multimodal large language model, supported by auxiliary modules such as voice activity detection (VAD) and text-to-speech (TTS) synthesis. The resulting system operates in a train-free, plug-and-play manner. Experiments on the HumDial dataset demonstrate the effectiveness of our framework, which ranks second among all teams on the test set of the Human-like Spoken Dialogue Systems Challenge [ 12 ] (Track 2: Full-Duplex Interaction). Code is available at the GitHub repository https://github.com/yu-haoyuan/fd-badcat .



†
†
address:

1
Hunan University
2
Gongdao Technology
3
Jilin University

1
Introduction

Real-world voice interaction remains a challenging task
[
2
]
[
10
]
. Full-duplex interaction offers a more natural communication paradigm, unlike half-duplex systems that prevent users and machines from speaking concurrently. Full-duplex interaction is implemented through two paradigms, end-to-end models
[
1
]
and cascaded pipelines
[
4
]
. Paralinguistic cues such as emotion and prosody
[
5
]
play a crucial role in spoken dialogue. End-to-end models process user and agent speech jointly and preserve acoustic cues, but require heavy data and training. Cascaded pipelines built on TTS, automatic speech recognition(ASR), large language model (LLM), and speaker verification(SV) are easy to deploy and integrate. However, their multi-stage design loses paralinguistic cues and adds latency.

Cascaded pipelines remain attractive in practice because their pluggable control modules support barge-in, backchannels, and flexible component integration. Recent advances in multimodal large language model (MLLM) allow the models to use acoustic cues directly, making cascaded pipelines simpler.

[
9
]
uses an LLM to predict Speak/Listen state tokens that explicitly govern turn-taking. EasyTurn
[
7
]
[
6
]
categorizes user utterances into four states: complete, incomplete, wait, and backchannel, making the model’s predictions more interpretable and accurate. FireRedChat
[
3
]
presents a practical pluggable full-duplex framework, and StepFun
[
11
]
further explores chronological thinking in end-to-end full-duplex spoken dialogue models to improve fine-grained speech handling.

In this work, we propose a new pluggable full-duplex dialogue framework which unifies turn-taking signals and user utterance states into a single decision space, enabling the MLLM to make coherent state-dependent decisions.
The framework is composed of four modular components:
(1) an audio acquisition module which employs VAD and SV to monitor the incoming speech stream;
(2) a context module which uses an asynchronous ASR to provide user transcripts;
(3) a decision module powered by a MLLM;
and (4) a speech generation module which synthesizes responses through TTS.
Relying on the MLLM’s built-in multimodal reasoning, the framework is train-free, and uses ASR transcripts only as contextual augmentation, thereby realizing a semi-cascaded design.
Experimental results show that the proposed framework achieves state-of-the-art semantic and interaction-state inference with reduced response latency on the HumDial dataset.

Figure 1:
Overview of the unit-based dialogue process. Top-left shows example MLLM control behaviors; bottom-left shows intra-unit listen/speak transitions driven by continue/switch signals. Right shows one unit’s execution, where kl (keep listen) / l2s (listen-to-speak) correspond to continue/switch in the listen state and ks (keep speak) / s2l (speak-to-listen) correspond to continue/switch in the speak state.

2
Method

Our framework decomposes natural dialogue into a sequence of dialogue units.
Each unit contains two states: listen and speak.
The system always begins a unit in the listen state, and the MLLM issues one of two actions—
continue
or
switch
—to drive state transitions.

Continue
keeps the system in the current state, while
switch
moves the system to the other state.
A switch from listen to speak occurs within the same unit, whereas a switch from speak back to listen marks the beginning of the next unit.
Through these two actions, the MLLM controls both state transition and unit progression.

To determine state transition, we simplify and reorganize the categories introduced in EasyTurn
[
7
]
.
In the listen state, the system determines whether the user’s utterance is semantically complete, with incomplete utterances outputting a continue decision and complete ones outputting a switch decision that moves the system to the speak state, where the system generates the assistant’s spoken response.
In the speak state, the system distinguishes between user backchannels and genuine interruptions, with backchannels outputting a continue decision and interruptions outputting a switch decision that returns the system to the listen state.

The audio acquisition module continuously captures the user’s speech stream, optionally applying speaker verification to filter non-target voices, and forwards the audio to the MLLM together with a state-specific prompt.
In parallel, the audio is sent to the context module, where an asynchronous ASR generates transcripts that are cached and used in the next decision cycle as semantic augmentation.
During the speak state, the TTS module renders the assistant’s response, and any user interruption immediately terminates playback and returns the system to the listen state to begin a new unit.

3
experiments

Our system is implemented using open-source components. The audio acquisition module uses Silero VAD
1
1

1

https://github.com/snakers4/silero-vad

to detect speech activity in real time, with an optional speaker-verification model CAM++
2
2

2

https://modelscope.cn/models/damo/speech_campplus_sv_zh-cn_16k-common

to identify and filter target speakers. The context module
uses Paraformer
3
3

3

https://github.com/k2-fsa/sherpa-onnx
, which provides ASR results as auxiliary semantic context. The multimodal decision module uses Qwen3-Omni
4
4

4

https://github.com/QwenLM/Qwen3-Omni

to integrate acoustic cues and contextual information for unit-level control decisions. The speech generation module adopts IndexTTS1.5
5
5

5

https://github.com/Ksuriuri/index-tts-vllm

to synthesize agent responses with streaming support.
Qwen3-Omni and IndexTTS 1.5 are vLLM-based, and the system prompts are available in our GitHub repository.
Experiments are conducted on two NVIDIA A100 GPUs using the dataset provided by the challenge organizers, with evaluation largely following Full-Duplex-Bench v1.5
[
8
]
and additional metrics for full-duplex assessment.

Table 1:
Experimental Results on the Dev and Test Set

First

Response

Delay

Interruption

Total

Score

Rejection

Total

Score

Total

Delay

baseline

2.753s

80.2

45.6

2.436s

ours(Dev)

1.528s

89.7

50.0

1.698s

ours(Test)

-

89.7

57.8

1.632s

We implement front-end and back-end interaction using WebSocket and FastAPI to simulate real-world dialogue. The experiments show that a MLLM can replace the traditional ASR-to-LLM cascade by reasoning directly over user audio, enabling the system to capture paralinguistic cues, reduce latency, and produce more informed responses.

4
conclusion

Read the complete text