Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition

Sean, Hong, Jihwan, Sudarsana Reddy, Dani, Louis, Shrikanth · 2025

Read Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition by Sean, Hong, Jihwan, Sudarsana Reddy, Dani, Louis, Shrikanth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24059.txt.

Description

Sean Foley, Hong Nguyen, and colleagues separate the contributions of mouth movement and sound in recognizing phonemes in a 2025 speech science study. Real-time magnetic resonance imaging films the vocal tract while a person speaks, giving audio and articulation together. Earlier multimodal work pooled many speakers, and the large differences between speakers' anatomy drowned the fine link between movement and sound.

Single-speaker depth lets the authors train separate audio-only and video-only models and compare them cleanly. Knowing which channel carries which phoneme distinctions guides hearing technology and speech therapy tools that teach articulation.

Catalog record

CREATORSean, Hong, Jihwan, Sudarsana Reddy, Dani, Louis, Shrikanth
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24059

Browse the full library · About the Tower of Babel