Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang, Bohan Zhuang Mingkui Tan · 2024
Read Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion by Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang, Bohan Zhuang Mingkui Tan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01289.txt.
Description
Zhuokun Chen and five co-authors present a way to sharpen the vision of multimodal language models in a 2024 paper. Such models join a vision encoder with a language model, and building a stronger encoder from scratch costs a lot of training. The authors note that models in one family tend to look at different parts of the same image.
Their method, VisionFuse, joins the visual tokens from several ready made models in one family and merges their language parts into one. No new training is needed. Tests across many tasks show clear gains, with one pairing lifting average scores by more than four percent and keeping deployment costs low.
Catalog record
| CREATOR | Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang, Bohan Zhuang Mingkui Tan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01289