Game-Time: Evaluating Temporal Dynamics in Spoken Language Models
Kai-Wei Chang \sthanks Co-first authors, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao-Hua Sun, Hung-yi Lee, James Glass · 2025
Read Game-Time: Evaluating Temporal Dynamics in Spoken Language Models by Kai-Wei Chang \sthanks Co-first authors, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao-Hua Sun, Hung-yi Lee, James Glass (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26388.txt.
Introduction
Kai-Wei Chang and co-authors test whether voice AI can keep time in conversation in this 2025 paper. Spoken language models promise real-time speech chat, yet timing skills like tempo and turn-taking went unmeasured. Their Game-Time benchmark starts with basic spoken instructions. Harder rounds add tempo rules and demand answers in sync.
Top models handled the basic tasks well, though many rivals failed even there. Under time pressure nearly every model fell apart. The results expose weak time awareness and shaky two-way interaction across the field. The authors release the tasks and demos as a base for building more fluent, time-aware voice systems.
Catalog record
| CREATOR | Kai-Wei Chang \sthanks Co-first authors, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao-Hua Sun, Hung-yi Lee, James Glass |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-26388
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Game-Time: Evaluating Temporal Dynamics in Spoken Language Models Kai-Wei Chang \sthanks Co-first authors, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao-Hua Sun, Hung-yi Lee, James Glass Abstract Conversational Spoken Language Models (SLMs) are emerging as a promising paradigm for real-time speech interaction. However, their capacity of temporal dynamics, including the ability to manage timing, tempo and simultaneous speaking, remains a critical and unevaluated challenge for conversational fluency. To address this gap, we introduce the Game-Time Benchmark , a framework to systematically assess these temporal capabilities. Inspired by how humans learn a language through language activities, Game-Time consists of basic instruction-following tasks and advanced tasks with temporal constraints, such as tempo adherence and synchronized responses. Our evaluation of diverse SLM architectures reveals a clear performance disparity: while state-of-the-art models handle basic tasks well, many contemporary systems still struggle with fundamental instruction-following. More critically, nearly all models degrade substantially under temporal constraints, exposing persistent weaknesses in time awareness and full-duplex interaction. The Game-Time Benchmark provides a foundation for guiding future research toward more temporally-aware conversational AI. Demos and datasets are available on our project website 1 1 1 https://ga642381.github.io/Game-Time . † † address: 1 Massachusetts Institute of Technology, USA 2 National Taiwan University, Taiwan 3 Academia Sinica, Taiwan 1 Introduction In the pursuit of human-like conversation with machines, the research frontier is moving beyond text-based Large Language Models (LLMs). The next challenge lies in mastering conversational dynamics in real-time speech, which has given rise to the field of conversational Spoken Language Models (SLMs) [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . This marks a critical shift from rigid turn-by-turn dialogues to fluid spoken interactions. Achieving such dynamics requires SLMs to operate in a real-time full-duplex manner [ 8 , 9 ] , where models must listen and speak simultaneously while producing seamless responses. This is inherently difficult, demanding synchronous speech generation, continual intent recognition, and precise control over both what to respond and when to respond. A further challenge lies in modeling the temporal dynamics of spoken interaction, where contemporary systems often fail to capture the fine-grained timing that is essential for advanced conversational fluency. For instance, they struggle to process user speech while planning a coherent reply that aligns with user-specified timing and tempo. This limitation underscores a fundamental deficiency: the lack of time-awareness . Figure 1: Overview of the Game-Time Benchmark , evaluating temporal dynamics in conversational Spoken Language Models (SLMs). Table 1: Game-Time Task Families. Basic Task contains fundamental tasks. Advanced Task contains temporal dynamics paired with Basic Task. \mathcal{N} : Number of subtasks. *The game of Rock paper scissors is itself an advanced task. Category Task Family \mathcal{N} Subtask / Paired Basic Tasks Description Basic 1 - Sequence 3 Number; Alphabet; Spell Generate sequential items in order from n_{\text{start}} to n_{\text{end}} . 2 - Repeat 2 Word; Sentence Repeat user-provided content \mathcal{C} . 3 - Compose 2 Word; Scenario Compose response including a target word w or fitting a scenario \mathcal{S} . 4 - Recall 3 Vocabulary; Letter; Rhyme Name N items that satisfy property \phi . 5 - Open-Ended 2 Empathy; QA Provide helpful and contextually appropriate content. 6 - Role-Play 2 Scenario; Persona Act within an imagined scenario \mathcal{S} or play a given persona \mathcal{P} . Adv. A - Time-Fast 10 [Multiple Basic Tasks] Complete the task quickly, within a specified duration \tau_{\text{fast}} . B - Time-Slow 7 [Multiple Basic Tasks] Perform the task slowly, taking at least the specified duration \tau_{\text{slow}} . C - Time-Silence 4 Repeat; Recall; Open-Ended Insert a silent interval of s seconds before the response. D - Tempo-Interval 4 Sequence; Recall Follow a specified tempo with \delta -second space between each word. E - Tempo-Adhere 4 Sequence; Recall Adhere to the tempo specified by the user’s spoken example \mathcal{C}_{\text{tempo}} . F - Simul.-Shadow 1 Repeat Repeat each word with immediate, word-by-word overlap. G - Simul.-Cue 1 Rock paper scissors* Overlap with the user by speaking at a designated timing or cue. Existing benchmarks for SLMs focus on content and style generation quality [ 10 , 11 ] , mimicking human dialogue behaviors (e.g. back-channeling) [ 12 ] , and turn-taking [ 13 ] . However, they lack the focus on temporal dynamics in the conversation. To address this critical gap, we introduce the Game-Time Benchmark , which is designed to assess the temporal capabilities of SLMs, focusing on their ability to perceive, predict, and produce speech in sync with the user. This work is inspired by childhood language acquisition [ 14 ] . Children learn to talk through language activities that require not only understanding the meaning of vocabulary, such as counting numbers and naming objects, but also acquiring a sense of timing and tempo in coordination games [ 15 , 16 ] . For example, playing “rock-paper-scissors” relies on sharing a common tempo and acting precisely on a specific cue (e.g., “shoot”). This inspiration led us to design the Game-Time Benchmark, which contains two categories of tasks. The Basic Tasks evaluate an SLM’s foundational ability to follow simple instructions, a challenge that can still prove difficult for modern SLMs. The Advanced Tasks build upon this foundation by augmenting the core instructions with temporal constraints. Here, the SLM must perform the basic tasks while fulfilling requirements for timing and synchronicity. The Game-Time Benchmark provides a framework for evaluating whether a model can move beyond mere content generation to acquire the temporal dynamics of conversational fluency. It proposes a novel perspective on evaluation, focusing not just on what to say , but critically, on when to say . In this paper, we evaluate various SLMs with different design philosophies, including Moshi [ 17 ] , Unmute [ 18 ] , Freeze-Omni [ 19 ] , Gemini-Live [ 20 ] , and GPT-realtime [ 21 ] . Our results show a performance disparity even on Basic Tasks: while state-of-the-art models generally excel, many contemporary SLMs still struggle with fundamental instruction-following. Furthermore, the performance of nearly all models degrades significantly when temporal constraints are introduced. Our findings indicate that models especially struggle with tasks requiring time awareness and real-time full-duplex capability, revealing a critical gap in the capabilities of current systems. For reproducibility, datasets and results are available on our website. 2 Related Works 2.1 Full-duplex Spoken Language Models Recent work has explored how SLMs can move beyond turn-based interaction toward full-duplex conversation [ 12 , 22 , 23 , 24 , 25 , 26 ] , where listening and speaking occur simultaneously. Two main modeling strategies have emerged to achieve full-duplex capability [ 3 ] : (1) Dual-channel SLMs [ 22 , 17 , 27 , 28 ] use two input channels: a listening channel for user speech and a speaking channel for the model’s own output. At each time step, the model simultaneously processes both inputs and generates output speech, which may be either voiced or explicitly silent. Although this architecture significantly increases modeling complexity, it naturally supports real-time listening and speaking concurrently. (2) Time-multiplexing SLMs [ 29 , 19 , 18 ] include a state prediction mechanism [ 30 ] that decides whether to speak or remain silent.