Re-Imagining Multimodal Instruction Tuning: A Representation View
Yiyang Liu James Chenhao Liang Ruixiang Tang Yugyung Lee Majid Rabbani Sohail Dianat Raghuveer Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han · 2025
Read Re-Imagining Multimodal Instruction Tuning: A Representation View by Yiyang Liu James Chenhao Liang Ruixiang Tang Yugyung Lee Majid Rabbani Sohail Dianat Raghuveer Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00723.txt.
Description
Liu and a large team present a study from twenty twenty-five on tuning multimodal models by editing their internal representations. Full fine-tuning of huge multimodal models costs too many parameters. Lightweight tuning methods lag behind in performance and stay hard to steer. Their method edits meaning-rich representations directly instead of weights. It needs a tiny fraction of tunable parameters. Scores beat leading baselines by wide margins in their tests. Editing special tokens inside the representations steers behavior simply.
The approach gives intuitive control over model conduct. Fewer moving parts make the tuning easier to read. The authors frame representation editing as a fresh tuning path. Gains hold across their multimodal benchmarks.
Catalog record
| CREATOR | Yiyang Liu James Chenhao Liang Ruixiang Tang Yugyung Lee Majid Rabbani Sohail Dianat Raghuveer Rao Lifu Huang Dongfang Liu Qifan Wang Cheng Han |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00723