Heterogeneous bimodal attention fusion for speech emotion recognition

Jiachen Luo, Huy Phan, Lin Wang, Joshua Reiss* · 2025

Read Heterogeneous bimodal attention fusion for speech emotion recognition by Jiachen Luo, Huy Phan, Lin Wang, Joshua Reiss* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06405.txt.

Description

Jiachen Luo, Huy Phan, Lin Wang, and Joshua Reiss present a 2025 paper on recognizing emotion in conversation from voice and words together. Sound and text each carry feelings, but they live at different levels: raw audio features are low-level, while text meaning is high-level, and that gap weakens fusion. The team design a framework to bridge it.

Their method first enriches audio features with context, then fuses the two streams with an attention system guarded by a gate that filters false matches. A contrastive step further ties the modalities together. On two standard conversation datasets, MELD and IEMOCAP, the approach beat previous best systems, showing that respecting the level gap between sound and text pays off.

Catalog record

CREATORJiachen Luo, Huy Phan, Lin Wang, Joshua Reiss*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06405

Browse the full library · About the Tower of Babel