Re-Imagining Multimodal Instruction Tuning: A Representation View

Yiyang Liu James Chenhao Liang Ruixiang Tang Yugyung Lee Majid Rabbani Sohail Dianat Raghuveer Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han · 2025

Read Re-Imagining Multimodal Instruction Tuning: A Representation View by Yiyang Liu James Chenhao Liang Ruixiang Tang Yugyung Lee Majid Rabbani Sohail Dianat Raghuveer Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00723.txt.

Description

Liu and a large team present a study from twenty twenty-five on tuning multimodal models by editing their internal representations. Full fine-tuning of huge multimodal models costs too many parameters. Lightweight tuning methods lag behind in performance and stay hard to steer. Their method edits meaning-rich representations directly instead of weights. It needs a tiny fraction of tunable parameters. Scores beat leading baselines by wide margins in their tests. Editing special tokens inside the representations steers behavior simply.

The approach gives intuitive control over model conduct. Fewer moving parts make the tuning easier to read. The authors frame representation editing as a fresh tuning path. Gains hold across their multimodal benchmarks.

Catalog record

CREATORYiyang Liu James Chenhao Liang Ruixiang Tang Yugyung Lee Majid Rabbani Sohail Dianat Raghuveer Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00723

Browse the full library · About the Tower of Babel