InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang · 2025

Read InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model by Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12368.txt.

Introduction

Yuhang Zang and eleven co-authors release an open judge for vision-language models in a 2025 paper. Big models that read images and text sometimes answer wrong with total confidence. Reward models can grade their answers and steer them better, but open multimodal graders were nearly nonexistent.

Their grader trains on a large preference collection spanning text, images, and video: homework help, document reading, math, and more. It tops the newest public benchmark for such graders and holds its own on text-only tests. Put to work, it tutored a chat model through reinforcement training, lifting its instruction following and image answers. The team offers it as shared infrastructure.

Catalog record

CREATORYuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12368

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, Kai Chen, Dahua Lin, Jiaqi Wang Abstract Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data. † † footnotetext: * indicates equal contribution. {\dagger} indicates interns at IXCLab, Shanghai AI Laboratory Figure 1: (a) To train the IXC-2.5-Reward, we construct a multi-modal preference dataset spanning diverse domains (e.g., natural scenes, text-rich, reasoning) and modalities (image, text, video). (b) The framework of IXC-2.5-Reward. (c) The IXC-2.5-Reward guides policy training for IXC-2.5-Chat via reinforcement learning. 1 Introduction “ If you don’t know where you are going, you’ll end up some place else. ” — Yogi Berra Reward Models (RMs) Cai et al. (2024) ; Zhu et al. (2023) ; Liu et al. (2024a) ; Wang et al. (2024f) ; Wang et al. (2024b) ; Yuan et al. (2024a) ; Lou et al. (2024) ; Yang et al. (2024b) ; Yuan et al. (2024b) ; Shiwen et al. (2024) ; Wang et al. (2024e) provide the crucial direction guidance about how well an AI model’s outputs align with human preference, and benefit Large Language Models (LLMs) in training and inference. During training, RMs are often used with reinforcement learning from human feedback (RLHF) Ouyang et al. (2022) ; Bai et al. (2022b) ; Schulman et al. (2017) ; Rafailov et al. (2024) to penalize undesirable model behaviors and encourage outputs that align with human values. At inference, RMs facilitate test-time scaling strategies Snell et al. (2024) ; Gulcehre et al. (2023) , such as selecting the best response from candidate outputs or providing step-by-step critiques for complex reasoning tasks Zelikman et al. (2022) ; Hosseini et al. (2024) . Despite their crucial role in both training and inference, multi-modal RMs for Large Vision Language Models (LVLMs) remain notably underexplored compared to language-only RMs for LLMs. Because current preference data is predominantly text-based and skewed toward specific domains (e.g., safety), data scarcity poses a significant challenge to training multi-modal RMs for diverse modalities such as images, videos, and text. Consequently, existing multi-modal RMs Wang et al. (2024a) ; Xiyao et al. (2024) are largely constrained to narrow domains (e.g., mitigating hallucinations) or rely on prompting LVLMs with evaluation prompts, effectively functioning as generative RMs Xiong et al. (2024) . The limitation of multi-modal RMs subsequently constrains the capabilities of open-source LVLMs such as instruction following and safety-should-refuse, thereby hampering user interaction experience in multi-modal chat scenarios. The growing community interest in RLHF and test-time scaling highlights the need for multi-modal RMs, which motivates us to present InternLM-XComposer2.5-Reward (IXC-2.5-Reward). Instead of directly transferring unimodal (text) reward models (RMs) to the vision modality, we augment the existing LVLM (InternLM-XComposer2.5) with an additional scoring head to predict reward scores. An effective multi-modal RM should ideally possess two key properties: (1) the ability to predict reward scores for both image, video, and textual inputs and (2) the capacity to generalize across diverse domains, such as instruction following, knowledge, text-rich images (e.g., documents), reasoning tasks, etc. To this end, we develop a pipeline (Fig. 1 (a)) to construct multi-modal preference data, and also incorporate existing high-quality datasets. This pipeline selects prompts across diverse domains for text, image, and video inputs, generates corresponding responses, and then uses GPT-4o Hurst et al. (2024) or verifier Lambert et al. (2024a) to perform preference judgments. Trained on our preference data, IXC-2.5-Reward effectively evaluates both visual (image and video) and textual inputs (Fig. 1 (b)). IXC-2.5-Reward achieves best performance on multi-modal VL-RewardBench Li et al. (2024b) (70.0%) that beat all previous generative RMs including Gemini-1.5-Pro (62.5%) and GPT-4o (62.4%). Even on uni-modal (text) RM benchmarks, IXC-2.5-Reward also demonstrates good results, with an average score of 88.6\% on Reward-Bench Lambert et al. (2024b) and 68.8\% on RM-Bench Liu et al. (2024b) . We further demonstrate the effectiveness of IXC-2.5-Reward in the following three aspects: (1) IXC-2.5-Reward for RL training. We train a chat model (IXC-2.5-Chat) using the on-policy Proximal Policy Optimization (PPO) algorithm with IXC-2.5-Reward to enhance its ability to follow instructions and provide a better user experience in multi-modal conversations. Our results show clear improvements of IXC-2.5-Chat on multi-modal instruction following and in-the-wild chatting benchmarks, which validate the effectiveness of IXC-2.5-Reward for providing the reward signal during RL training. (2) IXC-2.5-Reward for Test-Time Scaling. Using best-of- N sampling with IXC-2.5-Reward leads to additional performance gains compared to the RL-trained IXC-2.5-Chat, confirming IXC-2.5-Reward’s effectiveness in selecting good responses from candidate responses. (3) IXC-2.5-Reward for Data Cleaning. We observe a strong correlation between low IXC-2.5-Reward scores and problematic samples, such as those exhibiting hallucinations or mismatched image/video and question/answer content. This suggests that IXC-2.5-Reward can effectively clean LVLM pre-training and post-training data. Table 1: Overview of existing preference datasets used in IXC-2.5-Reward. Category Dataset Text IF General Tulu-3-IF-augmented-on-policy-8b Lambert et al. (2024a) UltraFeedback Cui et al. (2024) Safety hhh alignment Askell et al. (2021) , PKU-Safe Dai et al. (2024a) SHP Ethayarajh et al. (2022) , Anthropic-hhrlhf Bai et al. (2022a) Image Chat WildVision-Battle Lu et al. (2024c) General LLaVA-Critic Xiong et al. (2024) , VL-Feedback Li et al. (2024c) , RLAIF-V Yu et al. (2024b) MIA-DPO Liu et al. (2024e) Table 2: Overview of the source of newly collected data used in IXC-2.5-Reward. Category Dataset Image IF General MM-IFDPO-23k Ding et al. (2025) KVQA Shah et al. (2019) , A-OKVQA Schwenk et al. (2022) , PMC-VQA Zhang et al. (2023) Text-Rich AI2D Kembhavi et al. (2016) , IconQA Lu et al. (2021) , TQA Kembhavi et al. (2017) ChartQA Masry et al. (2022) , DVQA Kafle et al. (2018) , ScienceQA Lu et al. (2022a) Reasoning GeoQA Chen et al. (2021) , CLEVR-Math Lindström and Abraham (2022) Super-CLEVR Li et al. (2023) , TabMWP Lu et al. (2022b) Video General TrafficQA Xu et al. (2021) , FunQA Xie et al. (2024) , MiraData Ju et al. (2024) 2 Related Work Reward Model in Large Language Models. Reward models (RMs) are crucial for both Reinforcement Learning from Human

Read the complete text