Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing

Yuanhe Tian, Pengsen Cheng, Guoqing Jin, Lei Zhang, Yan Song · 2025

Read Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing by Yuanhe Tian, Pengsen Cheng, Guoqing Jin, Lei Zhang, Yan Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07086.txt.

Description

Yuanhe Tian and four co-authors present an AI approach that reads feelings from image and text together, in this 2025 paper. Machines that judge mood from posts must weigh words against pictures, and the two often clash: a cheerful caption can sit on a grim photo. Simple fusion of the two streams misses such conflicts.

Their method first encodes and aligns the two streams with pretrained models. It then splits each stream's code into a shared part, carrying the common feeling, and a private part, carrying cues found in only one stream. An attention step blends the parts into a soft prompt for a large multimodal model. Tests covered aspect-based sentiment, emotion analysis, and hateful meme detection. The method beat strong rivals on all three tasks, which the authors credit to keeping shared and private cues apart.

Catalog record

CREATORYuanhe Tian, Pengsen Cheng, Guoqing Jin, Lei Zhang, Yan Song
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07086

Browse the full library · About the Tower of Babel