Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization

Xinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yi Yang, Linchao Zhu · 2026

Read Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization by Xinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yi Yang, Linchao Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01483.txt.

Introduction

Seven authors led by Xinyu Qiu train one vision-language model to both answer questions and check its own answers. The usual route trains a separate verifier, which doubles cost at training and at run time. Their method, ADPO, folds verification into the same reinforcement learning policy that generates answers.

Two ideas drive it. A verification reward uses the average scores of right and wrong samples as a threshold, rewarding the model when its judgment matches the truth. A decoupled advantage then optimizes generation and verification separately, with token masks keeping their gradients apart. The reported gains are large: verification quality up by about a third, inference time cut by more than half, and accuracy up on math, general, and interface benchmarks. One model that grades itself is cheaper to run and, the authors argue, better calibrated about its own limits.

Catalog record

CREATORXinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yi Yang, Linchao Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01483

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization Xinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yi Yang, Linchao Zhu Abstract Parallel test-time scaling typically trains separate generation and verification models, incurring high training and inference costs. We propose Advantage Decoupled Preference Optimization ( ADPO ), a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. ADPO introduces two innovations: a preference verification reward improving verification capability and a decoupled optimization mechanism enabling synergistic optimization of generation and verification. Specifically, the preference verification reward computes mean verification scores from positive and negative samples as decision thresholds, providing positive feedback when prediction correctness aligns with answer correctness. Meanwhile, the advantage decoupled optimization computes separate advantages for generation and verification, applies token masks to isolate gradients, and combines masked GRPO objectives, preserving generation quality while calibrating verification scores. ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time, with significant gains of +2.8%/+1.4% accuracy on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7%/+1.0% step success rate on AndroidControl/GUI Odyssey. Figure 1 : Overview of ADPO. We build a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. The unified policy model provides reliable scoring that enables effective test-time scaling via best-of-N selection, significantly improving performance across multimodal tasks. 1 Introduction Test-time scaling enhances reliability and performance by spending more compute at inference time, which can generally be realized in two ways 26 ; 7 ; 25 . Serial test-time scaling enables LLMs to enhance problem-solving capabilities by sequentially generating additional thinking tokens 25 ; 39 . Parallel test-time scaling generates multiple solutions simultaneously, then aggregates candidates and selects the best one 38 ; 42 . Serial test-time scaling, as demonstrated by DeepSeek-R1 7 and OpenAI-o1 26 , achieves substantially improved performance in mathematics and coding domains. However, when transferring to multimodal domains, recent works have found that serial test-time scaling provides only limited performance improvements on image classification, video understanding and visual spatial understanding tasks 12 ; 14 ; 16 ; 46 ; 34 . These observations highlight the inherent limitations of serial test-time scaling and critically motivate the development of principles for parallel test-time scaling that more effectively support robust and efficient multimodal reasoning 43 ; 3 ; 40 . Current approaches 33 ; 5 deploy two separate models: a generator that creates potential solutions and a verifier that evaluates and ranks these candidates, leading to substantial performance gains. However, this methodology suffers from two major limitations: 1) training resource intensive , as it requires preparing two specialized datasets and training two independent models; 2) deployment inefficiency , since both models must operate concurrently during inference, demanding significant computational resources. Alternatively, training only a generator (with majority voting for selection) or only a verifier (using a base model for generation) yields limited performance improvements compared to training both components 33 ; 38 ; 4 . To address these limitations, we propose a novel reinforcement learning framework that trains a unified policy model to concurrently generate both solutions and self-verification scores of the solutions ( Fig. 1 ). We propose Advantage Decoupled Preference Optimization (ADPO), a unified framework that enables reliable self-verification while maintaining generation quality. This unified paradigm faces two challenges. First, training the model to verify its own outputs using binary rewards creates severe class imbalance. As the model improves, the proportion of correct answers increases substantially, causing verification scores to collapse toward uniform predictions and eliminating gradient signals ( Section 3.2 ). Second, naively aggregating answer and verification rewards leads to reward hacking. The model exploits the objective by deliberately producing incorrect answers while assigning low verification scores, achieving high total rewards while severely degrading generation performance. To address the class imbalance challenge, we introduce a preference verification reward that reframes verification as a ranking problem. Instead of comparing scores against fixed thresholds, our approach adaptively partitions samples into positive/negative groups based on answer quality, then rewards the model when verification scores respect the relative quality ordering within each group. By computing advantages from pairwise comparisons rather than absolute labels, this formulation maintains informative gradient signals even under severe class imbalance. To address the reward hacking challenge, we introduce advantage decoupled optimization that computes separate advantages for answer rewards and preference rewards. Specifically, answer advantages are estimated from answer rewards only, while preference advantages are derived solely from preference rewards. We then apply disjoint token-level masks to isolate gradient flows. In this way, answer advantages exclusively update generation tokens, while preference advantages solely affect verification score tokens. This decoupled computation ensures that improvements in generation quality are driven only by answer-based gradients, while verification calibration is shaped only by preference-based gradients, thereby eliminating reward hacking. Our contributions are summarized as follows: 1. Preference verification reward. We develop Preference Verification Reward, which maintains informativeness under severe class imbalance while improving calibration and robustness. 2. Advantage decoupled optimization. We introduce a principled approach to disentangle content generation and verification learning within a unified GRPO framework. 3. Comprehensive evaluation. Our method significantly improves task performance and verification quality: ADPO achieves +2.8/+1.4 accuracy gains on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7/+1.0 step success rates on AndroidControl/GUI Odyssey. 2 Related Work Test-Time Scaling. Recent work scales reasoning at test time via longer thinking tokens and majority voting for LLMs 7 ; 26 ; 38 ; 31 . Multimodal variants adapt this paradigm with R1-style objectives and structured CoT for VLMs 19 ; 28 ; 20 ; 32 ; 9 ; 47 ; 41 . In agentic settings, GUI agents adopt RL with explicit reasoning traces 24 ; 17 ; 29 ; 10 ; 50 ; 6 . However, recent “no-think” results suggest that more internal tokens do not always translate to better multimodal reasoning 12 ; 14 . We instead couple solution generation with a learned self-verification signal, enabling reliable performance scaling through best-of- N selection without fragile dependence on longer chains. Multimodal Reward Modeling and Generative Verifiers. Another line studies reward modeling for multimodal alignment, including RLHF-style pipelines and chain-of-thought verification 49 ; 33 . Process or scalar reward models provide step-level or outcome supervision for reasoning 5 ; 2 ; 36 ; 27 ; 4 ; 15 ; 8 . Generative verifiers and LLM-as-judge train models to both solve and judge 48 ; 51 . In contrast, we use reinforcement learning to train a single policy for answer and calibrated confidence with separate advantages and mutual masking, and we do not finely control the positive/negative ratio in training data; instead, we employ preference reward rather than binary reward, enabling dependable best-of- N across multimodal tasks. 3 Method We propose Advantage Decoupled Preference Optimization , a unified reinforcement learning framework that extends GRPO to jointly learn answer generation and self verification within a single policy. For each multimodal query, our model first generates an answer and then predicts a verification score. At inference, we use batch decoding to sample multiple candidate answers and select the one with the highest verification score as the final output. This unified generation and verification paradigm achieves reliable self verification and reduces inference latency. To realize this

Read the complete text