Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach

Changdae Oh, Zhen Fang, Shawn Im, Xuefeng Du, Yixuan Li · 2025

Read Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach by Changdae Oh, Zhen Fang, Shawn Im, Xuefeng Du, Yixuan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00577.txt.

Description

Changdae Oh and co-authors build, in a 2025 theory paper, the first formal frame for measuring a multimodal model's risk under data shift. Such models do well in matched settings but stumble when test data differ from training data. That gap is a safety concern for real use. Past work mostly ran tests without a shared risk measure.

Their frame rests on a new metric that scores how well a reply fits its prompt, across image and text together. They bound how much that metric can drop when data shift. The drop is tied to gaps in each sense. Tests across sixty-one shift cases on real benchmarks matched the theory well, the authors report.

Catalog record

CREATORChangdae Oh, Zhen Fang, Shawn Im, Xuefeng Du, Yixuan Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-00577

Browse the full library · About the Tower of Babel