Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty

Zhou Yang, Yueyi Yang · 2026

Read Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty by Zhou Yang, Yueyi Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00670.txt.

Description

Yang and Yang study how faces help us hear speech in this 2026 paper. When people talk face to face, we use sound plus the sight of the mouth, the eyes, and the whole face. Most speech systems use the mouth as the main visual cue. Feelings shown in the upper face are usually treated as a side task.

The pair train sentence classifiers on the CREMA-D corpus of acted speech. They test sound alone, then add mouth cues, upper-face cues, or both. Tests run on clean sound and on sound mixed with pink noise at three levels. Mouth cues help a lot when noise is high. Upper-face cues add little raw accuracy, the authors report. But full-face models stay better calibrated and beat shuffled-face controls in noise. Their claim is that shown feelings aid trust in what a system hears.

Catalog record

CREATORZhou Yang, Yueyi Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00670

Browse the full library · About the Tower of Babel