EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation
Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang · 2024
Read EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation by Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04903.txt.
Introduction
Yongxin Wang and co-authors present a method to improve models that answer questions about images in this 2024 paper. Such models can reason well but also make up facts. Training them on human-ranked answers helps, yet good ranking labels are costly to get.
Their method, called EACO, builds its own ranked data from only 5,000 images. A critic model is first trained to judge answers on several points, then picks better and worse answers for further training. A final round of plain training follows. The method cut made-up facts by about two thirds on one test and raised reasoning scores, with gains over a known base model across several tests.
Catalog record
| CREATOR | Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04903
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang Abstract Multimodal large language models (MLLMs) have achieved remarkable progress on various visual question answering and reasoning tasks leveraging instruction fine-tuning specific datasets. They can also learn from preference data annotated by human to enhance their reasoning ability and mitigate hallucinations. Most of preference data is generated from the model itself. However, existing methods require high-quality critical labels, which are costly and rely on human or proprietary models like GPT-4V. In this work, we propose E nhancing A lignment in MLLMs via C ritical O bservation ( EACO ), which aligns MLLMs by self-generated preference data using only 5k images economically. Our approach begins with collecting and refining a Scoring Evaluation Instruction-tuning dataset to train a critical evaluation model, termed the Critic. This Critic observes model responses across multiple dimensions, selecting preferred and non-preferred outputs for refined Direct Preference Optimization (DPO) tuning. To further enhance model performance, we employ an additional supervised fine-tuning stage after preference tuning. EACO reduces the overall hallucinations by 65.6% on HallusionBench and improves the reasoning ability by 21.8% on MME-Cognition. EACO achieves an 8.5% improvement over LLaVA-v1.6-Mistral-7B across multiple benchmarks. Remarkably, EACO also shows the potential critical ability in open-source MLLMs, demonstrating that EACO is a viable path to boost the competence of MLLMs. Figure 1 : Upper : Response examples from original LLaVA-v1.6 7B [ 25 ] and LLaVA-v1.6 7B w/ EACO, which mitigates hallucination and improve reasonging ability. Lower Left : The framework of EACO. The process begins with an image-question pair, which is fed to the initialized MLLM to genrate multiple responses. And then these responses are evaluated by a Critic Model that provides judgments regarding their quality. Based on the critic’s analysis, the responses are categorized into preference and non-preference groups. Finally, preference data is subsequently collected to further improve the MLLMs. Lower Right : Comparison with LLaVA-v1.6 7B [ 25 ] , LLaVA-RLHF [ 36 ] , Silkie [ 20 ] , SIMA [ 40 ] , and STIC [ 10 ] . Our proposed EACO framework achieves improvements across multiple metrics, demonstrating robust performance gains compared to other methods. 1 Introduction In recent years, Large Language Models (LLMs) have achieved remarkable success, largely driven by scaling up model size and enhancing data quality. In real-world applications, integrating inputs from other modalities, such as visual and auditory information, has propelled advancements in Multimodal Large Language Models (MLLMs) [ 26 , 25 , 19 ] . Existing MLLMs have demonstrated notable progress, especially within the research community [ 25 , 8 , 2 ] , excelling in a range of downstream multimodal tasks such as visual question answering and image captioning [ 13 , 29 , 7 ] . Despite these achievements, MLLMs still face critical challenges in reasoning and hallucination. For example, MLLMs sometimes generate descriptions that include counterfactual visual elements, reflecting inaccuracies in visual reasoning and comprehension [ 27 ] in Figure 1 . To mitigate reasoning errors and hallucinations, ongoing research has focused on several core strategies that enhance model performance across modalities. Creating high-quality annotated datasets for MLLMs [ 26 , 18 , 21 , 24 ] is one of the most popular and effective approaches to improve model accuracy, reduce hallucinations, and enhance reasoning capabilities. However, such method is also highly resource-intensive, often presenting significant costs and logistical challenges. Recently, fine-tuning methods like Direct Preference Optimization [ 34 ] (DPO) refine model responses by aligning them more closely with human preferences. Utilizing preference data to guide the model in selecting preferred responses over less preferred alternatives. This approach discourages outputs that may be factually incorrect or lack logical coherence. While most existing methods [ 36 , 20 ] rely on feedback from human annotators or proprietary models to generate preference data, these approaches are often costly and dependent on access to specialized resources [ 20 ] . We seek to address these limitations by developing a scalable, critic-guided preference alignment strategy that uses self-generated preference data, thus reducing reliance on external feedback sources while maintaining high alignment quality. One recent pre-print work LLaVA-Critic [ 43 ] also introduces the similar critic-based learning. The difference lies in that our EACO focuses on the self-enhancement in DPO training while LLaVA-Critic prioritizes building a generalist evaluator. Given that one application of LLaVA-Critic is to provide reward-signals for iterative DPO training, we provide the detailed discussion from the perspectives of data composition, scoring selection, and training objectives in Section 2.2 . To enhance reasoning abilities and reduce hallucinations in MLLMs, our work introduces a novel and economical critic-based method. Specifically, we collect and refine a critic dataset comprising 51,000 images and over 137,000 critic instructions. This dataset is designed to fine-tune the model, enabling it to assess and critique responses with greater accuracy. The optimization process starts with a seed model that undergoes training on the critic dataset. This initial model is progressively fine-tuned to develop its critic abilities, allowing it to evaluate responses across multiple dimensions, such as Relevance, Basic Elements, and Clarity. After incorporating critic guidance into the response generation, the Critic then selects preferred and non-preferred outputs, which are used to guide refined Direct Preference Optimization [ 34 ] (DPO) tuning. The DPO process aligns the model more closely with high-quality responses, reducing tendencies toward hallucination and improving overall reasoning capabilities. The primary contribution of this work is the development of a novel, critic-based training framework that enhances reasoning abilities and reduces hallucinations in MLLMs in an economical manner. Key contributions include: • We propose EACO , a novel critic-based approach that guides MLLMs toward generating more accurate, contextually relevant, and hallucination-free responses. • We refine a Large-scale Critic Dataset containing 51,000 images and over 137,000 detailed critique instructions. This dataset is designed to train the model’s “critic” abilities, enabling it to assess responses across multiple dimensions. • EACO achieves an 8.5% average improvement on multiple benchmarks over baseline, which indicates effectiveness of the critic-based framework, especially on reasoning task and hallucination mitigation. 2 Related Work 2.1 Multimodal Large Language Models (MLLMs) MLLMs [ 26 , 25 , 8 , 19 , 4 ] have dominated a wide range of multimodal tasks, achieving remarkable progress in vision-language understanding [ 26 , 13 , 17 ] , reasoning [ 29 ] and generation [ 41 ] . Most of them rely on pre-trained uni-modal models, where MLLMs use learnable projectors to connect visual encoders with language models. These projectors are typically categorized as either query-based, like Q-Former [ 19 ] in MiniGPT-4 [ 54 ] and Instruct-BLIP [ 9 ] , which use cross-attention to capture visual signals, or projection layer-based, as seen in LLaVA [ 26 , 25 ] and ShareGPT4V [ 5 ] , where linear projection layers or Multi-Layer Perceptrons (MLPs) map visual features into the input space of language models. In contrast, Fuyu-8B [ 3 ] and Gemini [ 37 ] involve end-to-end training without pre-trained components. For example, Fuyu-8B processes raw image patches directly and transforms them into embeddings via linear projection, bypassing the use of pre-trained visual