Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang, Bohan Zhuang Mingkui Tan · 2024

Read Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion by Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang, Bohan Zhuang Mingkui Tan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01289.txt.

Description

Zhuokun Chen and five co-authors present a way to sharpen the vision of multimodal language models in a 2024 paper. Such models join a vision encoder with a language model, and building a stronger encoder from scratch costs a lot of training. The authors note that models in one family tend to look at different parts of the same image.

Their method, VisionFuse, joins the visual tokens from several ready made models in one family and merges their language parts into one. No new training is needed. Tests across many tasks show clear gains, with one pairing lifting average scores by more than four percent and keeping deployment costs low.

Catalog record

CREATORZhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang, Bohan Zhuang Mingkui Tan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01289

Browse the full library · About the Tower of Babel