Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding

Zhengyi Zhao, Shubo Zhang, Zezhong Wang, Luyao Ye, Huimin Wang, Hanqi Yan, Binyang Li, Kam-Fai Wong, Yulan He · 2026

Read Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding by Zhengyi Zhao, Shubo Zhang, Zezhong Wang, Luyao Ye, Huimin Wang, Hanqi Yan, Binyang Li, Kam-Fai Wong, Yulan He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03604.txt.

Description

Zhao and co-authors teach vision-language models to read memes for intent, not just looks. Asked what a sarcastic post means, such models often describe the image instead. Standard tuning tangles literal content with pragmatic meaning. Surface details then pollute the final answer. Intent Projection separates the two signals inside one model.

An orthogonal projection module strips dominant single-modality directions from the fused representation. What remains is the pragmatic residual. An affect classifier tags the polarity gap for the decoder. The model also writes a structured reasoning chain. A contrastive reward punishes answers that merely restate the literal description. Across six multimodal benchmarks, the method beats open-source rivals. It closes much of the gap to proprietary models. Gains are largest where literal and intended meanings diverge most.

Catalog record

CREATORZhengyi Zhao, Shubo Zhang, Zezhong Wang, Luyao Ye, Huimin Wang, Hanqi Yan, Binyang Li, Kam-Fai Wong, Yulan He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03604

Browse the full library · About the Tower of Babel