Vision-Language Models Can Self-Improve Reasoning via Reflection

Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou Yang Liu · 2024

Read Vision-Language Models Can Self-Improve Reasoning via Reflection by Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou Yang Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00855.txt.

Description

Abstract Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked.

Catalog record

CREATORKanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou Yang Liu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00855

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision-Language Models Can Self-Improve Reasoning via Reflection Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou Yang Liu Abstract Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, R 3 V , which iteratively enhances the model’s V ision-language R easoning by R eflecting on CoT R ationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that R 3 V consistently improves multimodal LLM reasoning, achieving a relative improvement of 23% to 60% over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation. 1 1 1 Our code is available at https://github.com/njucckevin/MM-Self-Improve . 1 Introduction Humans often rely on intuitive Chain-of-Thought (CoT) to perform complex reasoning Ericsson and Simon (1980) . Previous studies have shown that this CoT capacity also emerges in Large Language Models (LLMs) Wei et al. (2022) . Through simple prompting or fine-tuning Cobbe et al. (2021) ; Kojima et al. (2022) ; Hsieh et al. (2023) , CoT enhances the reasoning performance of LLMs while providing insights into their decision-making process. Recently, OpenAI o1 further advances reasoning by producing long internal CoT sequences, taking LLMs intelligence to a new level. Figure 1: Results of Qwen-VL on TabMWP, a visual mathematical reasoning dataset. Qwen-VL exhibits weak zero-shot CoT reasoning performance, while our R 3 V iteratively self-improves, surpassing the GPT-distilled baseline by a large margin. While CoT reasoning has significantly advanced LLMs in textual domains, extending CoT to multimodal settings remains an open problem. Unlike the abundant, unsupervised text-based CoT in pre-training corpora Kojima et al. (2022) ; Wei et al. (2022) , multimodal CoT resources are scarce in the text-dominated internet collections Dai et al. (2023) , hindering the full realization of Multimodal LLMs’ (MLLMs) reasoning potential. Recent studies show that open-sourced MLLMs struggle to integrate visual cues into their reasoning process, resulting in weak CoT performance Zhang et al. (2024a) ; Shi et al. (2024) . Consistent with our observations in Figure 1 , CoT prompting provides minimal gains over direct prediction Chen et al. (2024a) and falls far behind GPT-4o. One potential solution is to construct multimodal CoT annotations for post-training; however, manual annotation is prohibitively expensive and hard to scale. This raises our first research question: can MLLMs self-improve the reasoning capabilities through bootstrapping on CoT samples? Orthogonal to fine-tuning on curated CoT annotations, relying solely on positive samples can lead to suboptimal policy due to insufficient exploration of reasoning paths. Inspired by human thinking, another promising direction involves learning from trial-and-errors Yuan et al. (2024) ; Song et al. (2024) , where mistakes are not failures but key opportunities to enhance reasoning. A few multimodal approaches use corrupted prompts to create negative samples for preference learning, aiming to improve image comprehension Wang et al. (2023) ; Deng et al. (2024) . However, these methods fail to generate reasoning-aligned positive and negative CoT solutions, making them unsuitable for complex multimodal reasoning tasks. Thus, it remains unaddressed: how can MLLMs efficiently learn from mistakes to improve their reasoning skills? To address the above two questions, this paper proposes R 3 V , a self-training framework that enables the model to R eflect on bootstrapped CoT R ationales, thereby strengthening its V ision-Language R easoning. Firstly, we leverage MLLM’s pre-existing but weak CoT ability to bootstrap both rationales and answers for a given question, enabling the collection of a large number of positive and negative solutions based on answer correctness. Secondly, we introduce a reflection mechanism on negative solutions to help the model learn from mistakes. Specifically, we design self-refine and self-select losses that guide the model to correct flawed rationales and derive the correct answer by comparing rationale candidates, respectively. The above synergistic process can be repeated, with improved samples boosting MLLM’s reasoning and the enhanced model further improving rationale generation. Additionally, through self-select training, our model can derive the superior solution from multiple samples, further boosting performance via test-time computation. We conduct experiments across a wide range of multimodal reasoning benchmarks, including charts, geometry, commonsense, science, mathematics, etc. R 3 V progressively enhances the reasoning ability of MLLMs, delivering a 23%-60% relative accuracy improvement compared to GPT distillation, and consistently outperforming the strong self-training baseline, STaR Zelikman et al. (2022) . Moreover, our test-time selection is robust and effective, consistently surpassing Pass@1 and majority voting, even in OOD scenarios. Our main contributions are as follows: • We introduce an iterative self-training framework R 3 V that leverages CoT bootstrapped by MLLM itself for self-improvement. To our knowledge, this is the first attempt to apply self-training in vision-language reasoning. • We propose learning from mistakes through self-reflection, with support for test-time computation to further improve reasoning performance. • We perform extensive evaluations across 6 different multimodal domains to validate the effectiveness of R 3 V . Our analysis reveals the key factors driving the success of multimodal self-training. 2 Related Work Vision-Language Reasoning Beyond the extensively studied unimodal reasoning Cobbe et al. (2021) ; Sun et al. (2023) , multimodal reasoning has recently attracted significant interest as an essential part of human intelligence Yue et al. (2024) ; Lu et al. (2023) . Although MLLMs perform well on general vision-language benchmarks Liu et al. (2024) ; Chen et al. (2024b) , integrating visual cues into the reasoning process poses unique challenges, especially for open-source models Zhang et al. (2024a) ; Chen et al. (2024a) . Several studies have explored using rationale datasets to fine-tune models and enhance visual-language reasoning capabilities. For example, Gao et al. (2023) ; Zhang et al. (2024b) augmented existing mathematical datasets with rationales using GPT distillation, while Yang et al. (2024) enhanced performance through manually collected CoT annotations. In this work, we advocate for MLLMs to self-improve, reducing reliance on resource-heavy rationale annotations. Self-Training Methods Self-training helps the model learn from its own generated outputs, reducing the need for labor-intensive human annotations Yuan et al. (2024) ; Chen et al. (2024c) . Prior works have focused on enhancing the reasoning capacity of LLM. The typical approach involves sampling multiple rationales and filtering positive and negative solutions based on the answers. The LLM is then fine-tuned on the positive samples Zelikman et al. (2022) ; Hosseini et al. (2024) ; Yuan et al. (2023) or improved using preference learning Wang et al. (2024b) ; Mitra et al. (2024) , such as DPO Rafailov et al. (2024) . Recent advances have also extended self-training to agents Song et al. (2024) and neural symbolic Xu et al. (2024) scenarios. In this paper, we pioneer the exploration of self-training in vision-language reasoning, investigate the failure of DPO in multimodal settings, and address these challenges

Read the complete text