Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains

Jesen Zhang, Ningyuan Liu, Kaitong Cai, Sidi Liu, Jing Yang, Ziliang Chen, Xiaofei Sun, Keze Wang, Sun Yat-sen University, Corresponding author: [email protected] · 2025

Read Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains by Jesen Zhang, Ningyuan Liu, Kaitong Cai, Sidi Liu, Jing Yang, Ziliang Chen, Xiaofei Sun, Keze Wang, Sun Yat-sen University, Corresponding author: [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22545.txt.

Introduction

Jesen Zhang and seven co-authors present SR-MCR, a label-free method for making multimodal models reason more honestly about images. Such models often sound fluent while their step-by-step reasoning wanders and loses touch with the picture, because training usually grades only the final answer. SR-MCR instead scores the reasoning process itself using five self-referential cues drawn from the model's own outputs.

The cues measure semantic alignment, lexical fidelity, non-redundancy, visual grounding, and step consistency, combined into a reliability-weighted reward. A critic-free reinforcement objective with confidence-aware cooling steadies training and discourages trivial or overconfident answers. Built on Qwen2.5-VL, the seven-billion-parameter version reached state-of-the-art average accuracy among comparable open models across visual benchmarks, and ablations credited each reward term and the cooling module.

Catalog record

CREATORJesen Zhang, Ningyuan Liu, Kaitong Cai, Sidi Liu, Jing Yang, Ziliang Chen, Xiaofei Sun, Keze Wang, Sun Yat-sen University, Corresponding author: [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22545

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains Jesen Zhang, Ningyuan Liu, Kaitong Cai, Sidi Liu, Jing Yang, Ziliang Chen, Xiaofei Sun, Keze Wang, Sun Yat-sen University, Corresponding author: [email protected] Abstract Multimodal LLMs often produce fluent yet unreliable reasoning, exhibiting weak step-to-step coherence and insufficient visual grounding, largely because existing alignment approaches supervise only the final answer while ignoring the reliability of the intermediate reasoning process. We introduce SR-MCR , a lightweight and label-free framework that aligns reasoning by exploiting intrinsic process signals derived directly from model outputs. Five self-referential cues—semantic alignment, lexical fidelity, non-redundancy, visual grounding, and step consistency—are integrated into a normalized, reliability-weighted reward that provides fine-grained process-level guidance. A critic-free GRPO objective, enhanced with a confidence-aware cooling mechanism, further stabilizes training and suppresses trivial or overly confident generations. Built on Qwen2.5-VL, SR-MCR improves both answer accuracy and reasoning coherence across a broad set of visual benchmarks; among open-source models of comparable size, SR-MCR-7B achieves state-of-the-art performance with an average accuracy of 81.4%. Ablation studies confirm the independent contributions of each reward term and the cooling module. 1 Introduction Multimodal reasoning must deliver not only correct answers but also explanations that are coherent and visually grounded. Yet recent MLLMs (e.g., LLaVA 32 , Qwen-VL 2 ) often drift in their intermediate steps—contradicting themselves, hallucinating evidence 28 ; 79 ; 43 ; 74 ; 71 ; 63 , or repeating trivial content. Existing alignment pipelines, including instruction tuning 50 ; 62 ; 72 and preference finetuning (DPO 38 ; 64 , RLHF 36 ; 44 ; 73 ), rely on costly human-labeled rewards or external evaluators 77 ; 70 . While effective, they remain brittle under domain shift and largely outcome-centric , supervising answers rather than the reasoning process 29 ; 7 ; 68 ; 69 . This outcome-centric focus leaves two issues unresolved: (i) Lack of intrinsic process reward : supervising only final answers leaves step coherence and visual grounding underconstrained; (ii) Cross-domain instability : single reward proxies (e.g., lexical overlap) miscalibrate signals across heterogeneous tasks, causing over-alignment to spurious patterns. At the same time, a single forward pass of an MLLM already exposes multiple measurable signals that correlate with “good” reasoning—semantic alignment, lexical fidelity, non-redundancy, grounding to visual evidence, and local step consistency 60 ; 66 ; 67 . These signals are inexpensive to compute, task-agnostic, and complementary in nature. Rather than relying solely on external preference labels, we ask: can we turn these process-level signals into a practical, intrinsic self-reward for multimodal reasoning? We posit that, once normalized and reliability-weighted, such signals can serve as a unified self-reward for both training and diagnosis 57 ; 65 ; 61 . Based on this observation, we develop Self-Rewarded Multimodal Coherent Reasoning (SR-MCR) , a unified framework that instantiates process-aware, label-free alignment for MLLMs. Given an image I , textual input x , and model outputs (\hat{y}_{a},\hat{y}_{t}) (final answer and reasoning trace), SR-MCR computes a normalized self-reward \mathcal{R}(I,x,\hat{y}_{a},\hat{y}_{t})=\sum_{k\in\{\mathrm{sem,lex,nr,vis,step}\}}\lambda_{k}\,\tilde{s}_{k},\hskip 20.00003pt\lambda_{k}=\frac{\exp(\alpha\,\mathrm{Relia}_{k})}{\sum_{j}\exp(\alpha\,\mathrm{Relia}_{j})}. Here \tilde{s}_{k}\!\in\![0,1] are min–max normalized scores for semantic similarity, lexical overlap, non-redundancy, visual grounding, and step-wise coherence. The term \mathrm{Relia}_{k} estimates each signal’s reliability (e.g., held-out correlation or inverse variance), producing adaptive weights \lambda_{k} that attenuate noisy or unstable signals and amplify consistent ones. Optimization. Instead of constructing human preference pairs, we directly treat this self-reward as the optimization target and fine-tune the model with a self-rewarded GRPO 42 objective. The policy is updated to increase the likelihood of high-reward generations while constraining drift from the base model via a KL term 41 ; 40 ; 55 ; 14 (Sec. 3 , Eq. 9 ). To further stabilize training under noisy self-rewards, we introduce a dynamic cooling weight (Eq. 8 ) that down-weights trivial or overconfident samples based on their normalized negative log-likelihood 52 ; 26 . Together, these components yield a simple, label-free, and process-aware alignment recipe that can be applied across diverse visual domains. Contributions. This work is positioned as a practical, process-centric alignment framework rather than a new RL algorithm. Our main contributions are: • We identify a key gap in multimodal alignment—the lack of an intrinsic, process-aware reward for coherent and visually grounded reasoning. • We introduce SR-MCR , which fuses five self-signals into a reliability-weighted reward and optimizes MLLMs via GRPO with a simple cooling scheme. • We provide a lightweight, reproducible recipe that improves accuracy and reasoning coherence over Qwen2.5-VL baselines, with ablations clarifying each component’s role. 2 Related Work Vision-Language Models. Large Vision-Language Models (VLMs) couple visual encoders (e.g., CLIP 37 ; 13 ; 12 ; 3 , ViT 11 ) with LLMs for multimodal understanding. CLIP benefits from large-scale contrastive pretraining that yields strong zero-shot transfer and alignment between images and text, but it can struggle with compositional queries, and inherits biases present in its training data. LLaVA demonstrated that visual instruction-following can be achieved with a lightweight projector and a two-stage pipeline. Subsequent models such as Qwen-VL 75 improved perception and multilinguality via larger data and refined architectures, while GPT-4V 35 advanced visual reasoning. InstructBLIP 10 further enhances cross-modal fusion through Q-Former 24 . These works establish the foundation of modern multimodal reasoning. Alignment Methods for VLMs. Early alignment of vision–language models typically begins with Supervised Fine-Tuning (SFT) on curated multimodal datasets 49 ; 25 ; 27 ; 31 . While SFT imparts task formats and basic instruction following, it struggles to model nuanced human preferences 9 and often overfits to annotator distributions. RLHF extends SFT by training on preference data with a learned reward model 78 ; 4 , improving subjective alignment but incurring substantial annotation cost and introducing potential bias from the reward model itself. In contrast, RLVR (Reinforcement Learning with Verifiable Rewards) replaces learned rewards with rule-based or verifier-derived correctness signals, enabling more transparent and scalable optimization for tasks with objectively checkable outcomes (e.g., VQA or grounded reasoning), and avoiding the failure modes associated with preference-model drift. Reinforcement Learning Algorithms. PPO (Proximal Policy Optimization) is widely used for RL-based alignment; it uses an actor–critic setup with advantage estimation, which is often via GAE, and a clipped surrogate objective to stabilize updates, but requires an additional value (critic) network and thus incurs significant compute and memory overhead. GRPO (Group Relative Policy Optimization) 42 removes the critic and constructs relative advantages by comparing rewards within sampled groups of completions, substantially reducing computational and memory cost while retaining strong alignment performance in preference-based generation tasks. In practice, PPO tends to give lower-variance updates at higher resource cost, whereas GRPO is a lightweight alternative that is more sensitive to group construction and reward noise. 3 Method Figure 1 : Overview of SR-MCR . Given an image–text input (I,x) , the policy \pi_{\theta} generates multiple responses, each scored by five process-level self-reward terms and an adaptive reliability estimator to form a mixed reward. SR-GRPO then favors high-reward, reliable outputs while suppressing trivial ones, updating \pi_{\theta} on top of the frozen base policy \pi_{0} . Following the motivation in Sec. 1 , we introduce Self-Rewarded Multimodal Coherent Reasoning (SR-MCR) , a training framework that enables multimodal large language models (MLLMs) to self-evaluate and self-improve their reasoning process without human preference labels 1 ; 8 . SR-MCR is built on three components: (i) a unified

Read the complete text