Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems
Ashish Hallur 1 , Thomas Thebaud 1 , Georgi Tinchev 2 , Venkatesh Ravichandran 2 , Laureano Moro-Velazquez 1 · 2026
Read Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems by Ashish Hallur 1 , Thomas Thebaud 1 , Georgi Tinchev 2 , Venkatesh Ravichandran 2 , Laureano Moro-Velazquez 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31055.txt.
Introduction
Hallur and four colleagues work on evaluating speech-to-speech AI agents. These are systems that talk back in a generated voice. Judging them needs speech-native measures of prosody and rhythm: pitch, speaking rate, and pausing. But pooled statistics from many speakers misjudge any single output. These traits shift with the speaker and with the state of the conversation.
The team drew on more than 4,000 hours of two-person English conversation. From it they built matched reference groups for pitch level, pitch expressivity, speech rate, articulation rate, and pause behavior. An output is scored by percentile against its closest matched human group. Outputs outside the central range get flagged. On held-out human speech, matched references flagged about the expected share of cases. Pooled references over-flagged, and the direction of each deviation stayed easy to read.
Catalog record
| CREATOR | Ashish Hallur 1 , Thomas Thebaud 1 , Georgi Tinchev 2 , Venkatesh Ravichandran 2 , Laureano Moro-Velazquez 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.31055
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems Thanks: This work was supported by the SPEAR project. Add full funding acknowledgments before submission. Ashish Hallur 1 , Thomas Thebaud 1 , Georgi Tinchev 2 , Venkatesh Ravichandran 2 , Laureano Moro-Velazquez 1 Abstract Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm. Because F 0 F_{0} , speaking rate, articulation rate, and pausing shift with model-predicted speaker traits and interaction state, pooled human statistics can be poorly calibrated for evaluating a particular output. Using 4000+ hours of dyadic English conversation from the Seamless Interaction dataset, we construct matched reference regimes for F 0 F_{0} mean, F 0 F_{0} expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. We then define a percentile-based evaluation protocol: extract the same metrics from an S2S output waveform, compare them to the closest matched human reference stratum, and report percentile deviations or 5th–95th percentile out-of-regime flags. On held-out human rows, pooled references over-flag state-conditioned F 0 F_{0} expressivity and rhythm, while matched references return flag rates closer to the nominal 10% and make deviation direction interpretable. These outputs serve as behavioral plausibility checks that complement, rather than replace, perceptual and user-centered evaluation. I Introduction Spoken conversation is a highly coordinated form of communication, where interlocutors manage turn-taking while continuously adapting prosody and timing to the unfolding interaction [ 32 , 22 ] . Despite this complexity, transitions are typically smooth and supported by systematic cues, including prosodic and syntactic structure [ 20 ] . Speech technology has progressed toward S2S AI agents capable of producing fluent spoken responses. Evaluation, however, is still dominated by text-based measures, task success, or subjective ratings, which do not directly quantify speech-native interactional behavior. Systems trained primarily on read speech often fail to reproduce properties characteristic of spontaneous dialogue [ 28 ] . Conversational speech differs from read speech across multiple dimensions, including prosodic range and stress realization [ 16 , 15 ] . Temporal organization varies with context and listener demands, shaping speaking rate and rhythmic structure [ 36 , 42 , 8 , 44 ] . These patterns align with accounts of dialogue as adaptive coordination, where speakers modify behavior to meet various contextual demands [ 10 , 43 ] . Prosodic variation, particularly fundamental frequency ( F_{0} ), encodes communicative intent, emotion, and discourse structure, and interacts with prominence and stress cues [ 4 , 29 , 45 , 33 , 34 ] . Temporal structure is central to conversational timing, with systematic patterns in pausing and speaking rate [ 13 , 31 , 30 , 19 , 24 , 25 ] . Recent work also shows that conversational speech behaviors are context dependent, motivating analyses that explicitly condition on interaction factors [ 43 ] . Prior work has often examined prosody and timing in isolation or at a limited scale, leaving a gap in large-scale, multidimensional reference regimes for conversational speech technology. This gap is limiting for spoken dialogue evaluation, where a system may fall inside a pooled range while still producing prosody or timing that is implausible for the relevant speaker profile or interactional state. We frame large-scale conversational distributions as an operational evaluation resource rather than as descriptive baselines alone. Using robust percentile-based F_{0} measures and rhythm metrics from over 4,000 hours of English dyadic conversation, we make four contributions. First, we construct matched reference regimes for conversational F_{0} mean, F_{0} expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. Second, we show that pooled references can obscure systematic shifts associated with model-predicted sex label, model-predicted age bin, arousal, and dominance [ 43 , 40 ] . Third, we define a simple evaluation procedure that extracts the same metrics from an S2S output waveform and reports percentile deviations or out-of-regime flags relative to the closest matched human reference stratum. Fourth, we provide reproducible reference tables and an extraction/comparison protocol for speech-native behavioral plausibility checks in spoken dialogue systems. II Methods II-A Dataset We analyze dyadic conversational speech from the Seamless Interaction dataset, a large-scale corpus of face-to-face interactions designed to capture both Naturalistic interactions with untrained participants and Improvised interactions with trained actors. The released dataset contains 4,065.04 hours of interaction time, comprising 64,739 interactions segmented from 5,098 one-hour recording sessions involving 4,284 participants [ 1 ] . We use “interaction” to refer to a single conversational segment within a session. Each interaction yields two speaker channels, and all metrics are computed at the channel level. The dataset includes Naturalistic interactions with untrained participants and Improvised interactions with trained actors, supporting analyses that condition on interaction factors [ 10 , 43 ] . II-B Prosodic Metrics Fundamental frequency ( F_{0} ) is defined only during voiced speech, so each audio signal is analyzed frame-by-frame using the autocorrelation-based F_{0} estimator in Praat via parselmouth [ 18 ] . F_{0} extraction uses conservative bounds (75–500 Hz) to cover typical adult conversational ranges while reducing octave errors and spurious tracks [ 41 ] . We define the voiced ratio as the proportion of frames assigned a nonzero F_{0} value and exclude speaker-channels with a voiced ratio <0.05 [ 38 ] . To reduce sensitivity to spontaneous-speech artifacts and F_{0} -tracking outliers, we compute percentile-trimmed summaries by retaining voiced F_{0} values between the 10th and 90th percentiles within each speaker-channel [ 26 ] . We report the 10–90% trimmed mean, standard deviation, and range of F_{0} . II-C Temporal Metrics Temporal metrics are computed from word-level timestamps in the dataset’s ASR-aligned transcripts and the Voice Activity Detection (VAD) segments distributed with the corpus [ 1 ] . To obtain stable long-term estimates of rate, we follow evidence that speaking-rate estimates stabilize over an average stabilization time of 12.1 s (most values between 7.9 and 16.2 s) [ 2 ] . Accordingly, we define speech-activity stretches using the provided VAD, merge adjacent segments separated by at most 1.0 s, and retain only continuous stretches with duration \geq 12.1 s before computing speaker-channel-level temporal statistics. We define pauses as inter-word gaps \geq 0.2 s, a common practical threshold because shorter silences are difficult to distinguish from stop closures and their inclusion increases annotation and measurement burden [ 6 ] . Let W be the number of retained words, T the total duration (sum of retained stretch durations), and P the total pause time (sum of inter-word gaps \geq 0.2 s within stretches). In semi-spontaneous speech, WPM showed a very strong correlation with naïve listeners’ tempo ratings [ 17 ] , and hence we report speaking rates in words per minute (WPM) because it closely tracks perceived speech tempo and is directly measurable from word-aligned timestamps in the dataset. We report speech rate =60\cdot\frac{W}{T} and articulation rate =60\cdot\frac{W}{(T-P)} [ 12 ] in WPM, along with pause ratio =\frac{P}{T} . TABLE I: Pooled operating regimes (speaker-channel level). For each track, N and interaction-hours are constant across metrics and are reported in the header row. Metric Median IQR (25–75%) Mean Prosody ( N{=}121{,}813 , 3{,}863 h) F_{0} Mean [Hz] 157.4 120.1–198.6 161.5 F_{0} SD [Hz] 20.84 13.79–30.07 23.22 F_{0} Range [Hz] 87.11 57.11–125.8 95.82 Temporal ( N{=}91{,}471 , 3{,}045