TiCo: Time-Controllable Training for Spoken Dialogue Models
Kai-Wei Chang ♠ Wei-Chih Chen ♢ En-Pei Hu ♢ Hung-yi Lee ♢ James Glass ♠ ♠ MIT ♢ NTU [email protected] · 2026
Read TiCo: Time-Controllable Training for Spoken Dialogue Models by Kai-Wei Chang ♠ Wei-Chih Chen ♢ En-Pei Hu ♢ Hung-yi Lee ♢ James Glass ♠ ♠ MIT ♢ NTU [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22267.txt.
Introduction
Chang and co-authors present TiCo, a spoken dialogue model that obeys time limits. A user can ask for a 15 second reply, and TiCo aims to match that length. Most spoken models lack time awareness and miss such targets. The team builds TiCo-Bench to measure this skill. TiCo plants time markers as it speaks, so it can track elapsed time and adjust the rest. Training uses self-generation plus reinforcement learning, with no paired question-answer data. TiCo cuts duration error sharply against its backbone.
Catalog record
| CREATOR | Kai-Wei Chang ♠ Wei-Chih Chen ♢ En-Pei Hu ♢ Hung-yi Lee ♢ James Glass ♠ ♠ MIT ♢ NTU [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62054 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22267
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TiCo: Time-Controllable Training for Spoken Dialogue Models Kai-Wei Chang ♠ Wei-Chih Chen ♢ En-Pei Hu ♢ Hung-yi Lee ♢ James Glass ♠ ♠ MIT ♢ NTU [email protected] Abstract We propose TiCo , a simple post-training method for enabling spoken dialogue models (SDMs) to follow time-constrained instructions and generate responses with controllable duration. This capability is valuable for real-world spoken language systems such as voice assistants and interactive agents, where controlling response duration can improve interaction quality. However, despite their strong ability to generate natural spoken responses, existing models lack time awareness and struggle to follow duration-related instructions (e.g., “Please generate a response lasting about 15 seconds”). Through an empirical evaluation of both open-source and commercial SDMs, we show that they frequently fail to satisfy such time-control requirements. TiCo addresses this limitation by enabling models to estimate elapsed speaking time during generation through Spoken Time Markers (STM) (e.g., <10.6 seconds> ). These markers help the model maintain awareness of time and adjust the remaining content to meet the target duration. TiCo is simple and efficient: it requires only a small amount of data and no additional question-answer pairs, relying instead on self-generation and reinforcement learning. Experimental results show that TiCo significantly improves adherence to duration constraints while preserving response quality. Figure 1 : Overview of TiCo , a two-stage framework for time-controllable speech generation. Stage 1 (top): The model leverages self-generation to produce responses annotated with Spoken Time Markers (STMs) , which serve as supervision for learning time awareness , i.e., associating intermediate generation states with temporal progress and estimating elapsed speaking time. Stage 2 (bottom): The model is further optimized via RLVR, where rewards are derived from STMs, enabling the model to regulate response duration in real time. 1 Introduction “Time is money,” as famously stated by Benjamin Franklin, highlights the fundamental value of time in human life. In human–computer interaction, time is a critical resource that directly impacts usability, deployment cost, and safety-critical decision making. This is especially true for Spoken Dialogue Models (SDMs) [ 4 , 16 , 29 ] , which are gaining increasing attention in real-world applications, such as personal assistants, wearable devices, and healthcare systems [ 17 , 1 ] . In these scenarios, a response must not only be accurate and natural, but often also strictly bounded in duration. For example, a voice assistant may be required to provide a traffic update while driving; a wearable device may require concise spoken feedback due to battery or bandwidth constraints. Similarly, in medical or emergency scenarios, a voice assistant may need to deliver brief yet informative instructions under strict time pressure. In all of these cases, the ability to control response duration is a key requirement for practical deployment. Despite its importance, time controllability remains largely underexplored in SDMs. In the domain of text Large Language Models (LLMs), prior studies have shown that models often struggle to follow explicit length-constraint instructions [ 73 ] . Moreover, LLM outputs often exhibit verbosity or length bias, a phenomenon associated with preference-based evaluation and alignment [ 20 , 28 ] . This tendency weakens instruction-following capability and negatively affects user experience. While benchmarks, prompting and training strategies have begun to address length controllability in text LLMs [ 73 , 63 , 30 , 54 ] , this research direction remains active and continues to attract attention due to its substantial practical importance. However, controlling response duration in SDMs is considerably more challenging than controlling output length in text. In speech generation, word count is only a proxy for actual duration. A single word may contain different numbers of syllables, and speech duration is known to vary with phonetic composition, linguistic context, and prosodic structure [ 31 ] . Moreover, speaking rate may vary across speakers, speaking styles, and communicative conditions, which depends on both speaker and listener [ 36 ] . As a result, simply constraining the number of generated words does not guarantee accurate control over the final speech duration. This mismatch makes duration control a unique and more demanding problem in spoken dialogue systems. Given the limited study of time controllability in SDMs, we first introduce TiCo-Bench , a benchmark designed to evaluate the time-controllable instruction-following capability of SDMs. Our evaluation reveals that existing SDMs struggle to reliably satisfy explicit time constraints. To address this challenge, we propose TiCo (Figure 1 ), a simple yet effective post-training framework that enables SDMs to estimate and regulate generated speech duration in real time through Spoken Time Markers . The core idea of TiCo is to equip the model with an internal mechanism for time awareness during generation, allowing it to track temporal progress during generation and adjust its responses accordingly. Specifically, TiCo consists of a two-stage training procedure. In the first stage, the model leverages self-generation to construct supervision data for learning duration awareness, enabling it to associate intermediate generation states with temporal progress and estimate the elapsed speaking time. In the second stage, Reinforcement Learning with Verifiable Rewards (RLVR) [ 52 ] is applied, where rewards are automatically verified based on the Spoken Time Markers, to further shape the response distribution and improve compliance with duration-related instructions. This stage encourages the model to better satisfy target time constraints while preserving the response quality, including helpfulness and coherence. Our contributions are summarized as follows: • We propose TiCo , a two-stage training framework that enables spoken dialogue models to generate Spoken Time Markers (STMs) during inference and perform real-time control over response duration. • We introduce TiCo-Bench , the first benchmark designed to evaluate the time controllability of spoken dialogue models (SDMs), measuring whether they can follow explicit duration-related instructions. • We conduct extensive experiments showing that TiCo significantly improves duration controllability while preserving response quality, and further demonstrate that the learned capability generalizes beyond the duration range seen during training. 2 Related Works 2.1 Spoken Dialogue Models Spoken Dialogue Models (SDMs) [ 4 , 16 ] aim to enable natural human-computer interaction by directly understanding and generating spoken conversations. Unlike traditional voice assistants that rely on cascaded ASR, text generation, and TTS modules, recent SDMs increasingly adopt end-to-end or tightly integrated modeling paradigms [ 24 , 29 ] . However, compared to text-based LLMs operating in a semantically rich textual space, speech is consider to be significantly more challenging to process due to the high variability and complexity of acoustic signals 1 1 1 This challenge has been larely explored in prior work such as the “Textless NLP” paradigm [ 52 , 47 , 27 , 11 , 12 , 48 ] , where quantized speech representations are treated as “pseudo text” to improve training efficiency and efficacy. . As a result, many recent SDMs introduce intermediate representations , most commonly text, to support semantic planning during generating the speech response. This includes reasoning to improve response quality [ 15 ] , tool calling [ 5 ] to leverage external modules, and more direct guidance over spoken content [ 66 , 67 ] . Specifically, the SDM first takes the input query (in either text or speech form) to generate an intermediate representation, which is then consumed by a speech generator to produce the final output speech representation