Real-Time Voice AI Hears but Does Not Listen

Martijn Bartelds, Federico Bianchi, James Zou · 2026

Read Real-Time Voice AI Hears but Does Not Listen by Martijn Bartelds, Federico Bianchi, James Zou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26083.txt.

Introduction

Martijn Bartelds, Federico Bianchi, and James Zou test whether real-time voice AI listens to how callers sound. Four production systems face the trials: models from OpenAI, Google, and Alibaba. Each scenario makes tone carry the real message. Words and delivery point in opposite directions. The systems end calls with crying callers insisting nothing is wrong. They approve transfers authorized in frightened voices. They enrol callers whose agreement drips with sarcasm.

Perception is rarely the failure. Asked directly, three systems name the distress, fear, or sarcasm correctly. They then ignore it when deciding. The authors call this split the emotional intelligence gap. Accent and age estimates show the same pull toward word content over sound. Prompting the systems to attend to delivery helps only partly and unevenly. Until the gap closes, they warn, voice AI behaves like a transcript reader in settings where tone matters most.

Catalog record

CREATORMartijn Bartelds, Federico Bianchi, James Zou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26083

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time Voice AI Hears but Does Not Listen Martijn Bartelds, Federico Bianchi, James Zou Abstract Speech conveys information through both words and vocal delivery. We evaluate four leading production realtime voice systems—OpenAI’s GPT Realtime 2, Google’s Gemini 3.1 Flash Live, and Alibaba’s Qwen3.5 Omni Plus and Omni Flash—on tasks where the words and the delivery patterns both convey meaningful information. Across three consequential scenarios, all four systems act on the words rather than the voice. They end calls with crying callers who insist nothing is wrong, approve wire transfers authorized in frightened voices, and enroll callers whose agreement is clearly sarcastic. Surprisingly, this is often not a failure of perception. When asked directly, three of the four systems reliably identify the distress, fear, or sarcasm they later ignore when making decisions. We observe a similar pattern when these realtime voice systems estimate accent and age, as their responses frequently follow the biases of the words rather than the acoustic properties of the speaker. We term this disconnect between perception and action the emotional intelligence gap of voice AI. Prompting systems to explicitly attend to vocal delivery improves performance only partially and inconsistently. Our findings show that current realtime voice AI systems often behave as if speech had been reduced to a transcript, suggesting that they should be used with caution in settings where the tone and emotion of delivery convey important information. https://real-time-voice.github.io 1 Introduction When people speak, they convey information not only in the words they choose but also in how they say them. The lexical channel is the verbal content of an utterance. The non-lexical channel is everything the voice adds to it, such as pitch, tone, accent, and emotional state ( 17 ; 18 ) . Listeners draw on both, as both channels carry complementary information. In emergency medical calls, for example, a caller’s emotional distress is associated with patient outcomes, and taking this information into account can help improve medical triage ( 12 ) . Where the voice carries information in this way, a decision based on the words alone can be the wrong one. For instance, a distressed (i.e., the non-lexical cue) caller that says “everything is fine” (i.e., the lexical cue) should not be ignored. Figure 1: In each scenario the caller’s wording and delivery point to opposite actions, so the expected action (third column) turns on the delivery. The realtime voice systems tend to do the opposite (fourth column), acting on the wording and against the delivery. As voice becomes an everyday way of interacting with AI systems, it is critical to understand how these systems treat and act on lexical and non-lexical cues. In this paper, we study realtime voice systems , which are models that take speech as input and return speech as output in a live, turn-by-turn exchange, rather than cascaded systems that transcribe the speech, reason over the text, and synthesize a reply. We evaluate the leading production systems, which already power deployed voice agents, including in regulated settings such as healthcare ( 1 ) . How a system of this kind treats the non-lexical channel has not yet been established. We show that when the words and the voice of an utterance point to different conclusions (a distressed voice that says “everything is fine”), the four leading production realtime voice systems we study, OpenAI’s GPT Realtime 2 ( 13 ) , Google’s Gemini 3.1 Flash Live ( 8 ) , and Alibaba’s Qwen3.5 Omni Plus Realtime and Qwen3.5 Omni Flash Realtime ( 16 ) , act on the meaning they extract from the words and disregard the delivery. We find this across three consequential scenario-based decisions. In an emergency dispatcher welfare callback, every system ends the call on a caller who insists that nothing is wrong while crying. In a wire-fraud check, every system approves a wire transfer authorized in an audibly frightened voice as readily as one authorized calmly. In a volunteer recruitment call, every system signs the caller up, whether the agreement is spoken in earnest or in a mocking, sarcastic voice. We then probe the same conflict in single-turn diagnostics to better understand the behavior of these systems. When asked directly whether a speaker sounds distressed, three of the four systems answer yes far more often for a crying delivery than for the same words spoken calmly. The distress they disregard in the callback is one they can hear but choose not to act on. The fourth system, Qwen3.5 Omni Flash, misjudges delivery even when asked directly, yet acts on the words just the same. So, the same behavior appears whether or not the delivery is perceived. This finding is not limited to the tone of the delivery. When these systems are asked to name a speaker’s accent or age from a recording whose wording points to a different answer, the systems mostly base their answer on what the wording suggests. Human listeners recover both properties from the same recordings, and one system, Qwen3.5 Omni Plus, names several voices’ accents, so the cue is present and at least partly recoverable in a realtime pipeline. The move from text to speech has given these systems the speaker’s voice alongside the words, yet their decisions rest on the words alone, as if the voice had been reduced to its transcript. We term this asymmetry the emotional intelligence gap of voice AI. Wherever delivery rather than wording carries the decisive information, as in the emergency and security interactions modeled here, closing this gap is a precondition for deploying these systems safely. 2 Related work AI models have made significant progress on speech and general audio benchmarks in the last few years ( 5 ; 7 ; 10 ; 20 ) . Yet, several recent studies have shown that audio and speech language models rely on the words and discount the voice, most clearly when the two are put in conflict. In a benchmark of ten non-lexical, paralinguistic tasks, a synthesized voice contains one attribute while the transcript has another, and models recover the voice’s attribute poorly, returning instead the attribute the words contain ( 15 ) The same reliance has been shown for emotion. When the words carry no emotion, models predict neutral and ignore the emotion in the voice. Also, when the words and voice conflict, as in sarcasm, models register the conflict but cannot identify the specific emotion ( 2 ) . A separate study of four spoken language models reports the same pattern, showing that emotion predictions track the words far more strongly than the prosody, even when the prompt instructs them to judge from prosody alone and ignore the words ( 4 ) . It also extends to speaker identity, where models asked who said what in a dialogue perform about as well as a transcript-only system, recovering what was said while missing the voice that said it ( 19 ) . Two different causes have been proposed for the voice-word mismatch observation. One traces the bias to the models’ origin in text-only language backbones adapted through later multimodal fine-tuning, which can carry over a preference for the words ( 2 ) . The second explanation is architectural. In models that pair an audio

Read the complete text