Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought

Yunheng Li*, Hangyi Kuang*, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou † {\dagger}, Ming-Ming Cheng · 2026

Read Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought by Yunheng Li*, Hangyi Kuang*, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou † {\dagger}, Ming-Ming Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22847.txt.

Introduction

Li and co-authors rethink token-level training for multimodal chain-of-thought reasoning. Common reward methods treat all reasoning tokens the same. Yet tokens differ in how much they ground vision or explore ideas. The authors study token patterns in successful reasoning traces. Their method, PEPO, builds a perception prior from hidden state similarity. It blends that prior with token entropy to set token-level advantages. PEPO plugs into GRPO and DAPO with no extra supervision. Tests across geometry, grounding, puzzles, and few-shot tasks show steady gains.

Catalog record

CREATORYunheng Li*, Hangyi Kuang*, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou † {\dagger}, Ming-Ming Cheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56812
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22847

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought Yunheng Li*, Hangyi Kuang*, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou † {\dagger}, Ming-Ming Cheng Abstract Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories that interleave perceptual grounding with multi-step inference. However, existing Reinforcement Learning with Verifiable Rewards (RLVR) methods typically optimize reasoning at a coarse granularity, treating CoT uniformly without distinguishing their varying degrees of visual grounding. In this work, we conduct a token-level analysis of multimodal reasoning trajectories and show that successful reasoning is characterized by structured token dynamics reflecting both perceptual grounding and exploratory inference. Building upon this analysis, we propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior from hidden state similarity and integrates it with token entropy through a smooth gating mechanism to produce token-level advantages. PEPO integrates seamlessly with existing RLVR frameworks such as GRPO and DAPO, requiring neither additional supervision nor auxiliary branches. Extensive experiments across diverse multimodal benchmarks demonstrate consistent and robust improvements over strong RL baselines, spanning geometry reasoning, visual grounding, visual puzzle solving, and few-shot classification, while maintaining stable training dynamics. 1]VCIP, School of Computer Science, Nankai University 2]Kuaishou Technology ] ∗ Equal Contribution. † Corresponding author. \mclink [Project Page] https://github.com/xzxxntxdy/PEPO 1 Introduction Large Vision-Language Models (LVLMs) [ 3 , 88 , 64 , 21 , 58 , 75 ] have achieved impressive progress across diverse vision-language tasks, such as question answering [ 2 , 18 ] , visual reasoning [ 36 , 83 , 45 , 46 ] and reasoning grounding [ 25 , 78 , 27 ] . Recent advances in LVLMs [ 51 , 71 , 38 , 33 , 32 , 7 , 66 , 89 , 80 ] have focused on enhancing their reasoning capability, where Reinforcement Learning (RL) serves as an effective way to optimize Chain-of-Thought (CoT) [ 68 ] reasoning and improve performance. Figure 1 : Overview of PEPO. (a) Effective multimodal reasoning arises from the complementarity between perception and exploration. Abbr. Exp.: Exploration-only, Per.: Perception-only, P+E: Perception + Exploration. (b) Unlike traditional sequence-level optimization with uniform advantages, PEPO reweights tokens using a perception prior from visual similarity and entropy via a smooth gate, producing fine-grained token-level advantages. (c) When integrated with GRPO or DAPO, PEPO consistently improves performance across diverse benchmarks. Typical LVLM training pipelines incorporate RL with verifiable rewards to refine CoT reasoning, commonly under frameworks such as Group Relative Policy Optimization (GRPO) [ 19 , 50 ] . For example, most approaches adopt outcome-based rewards (e.g., format or accuracy) under the assumption that improving answer format or textual correctness naturally leads to coherent reasoning. However, these methods suffer from sequence-level supervision, which fails to distinguish the contributions of intermediate CoT steps. To mitigate this, several works in Large Language Models (LLMs) introduce token-level entropy advantages to encourage exploration at uncertain CoT steps [ 62 , 8 , 63 ] . Nevertheless, entropy-based advantages mainly capture textual uncertainty but show weak correspondence to visual semantics and insufficient discrimination of reasoning relevance. Recent perception-aware RL methods incorporate visual signals, but often introduce additional computational overhead through auxiliary masking branches [ 67 , 20 ] or attention-based measures [ 22 ] that are incompatible with efficient acceleration frameworks [ 12 ] . Unlike text-only LLMs, LVLMs reason under multimodal constraints, where visual perception and exploratory dynamics play complementary roles in shaping the CoT process, as illustrated in \figref fig:intro(a). From the perceptual perspective, token-level analysis in \secref sec:analysis reveals that correct reasoning is strongly associated with perceptual grounding: accurate responses consistently depend on a compact subset of visually aligned tokens that anchor the CoT process. More importantly, a simple hidden-state similarity between response and visual tokens captures this association, providing a modality-specific indicator within the fine-grained reasoning process and reflecting linguistic-perceptual alignment. Complementing perception, token-level entropy from the logits highlights uncertain steps where alternative reasoning paths should be explored. However, we find that existing RLVR frameworks overlook the fine-grained coupling between perceptual grounding and reasoning dynamics, relying mainly on outcome- or entropy-based supervision, as well as mask-based perception-aware methods that fail to capture modality-specific interactions. Motivated by the aforementioned analysis, we introduce Perception-Exploration Policy Optimization (PEPO), a token-level policy optimization framework that couples visual perception and exploration to enhance CoT reasoning in LVLMs. The core of PEPO is to convert hidden-state similarity into a calibrated perception prior without auxiliary branches or additional supervision. Specifically, for each response token, we compute cosine similarity between its hidden state and the set of visual-token states and aggregate it into a per-token visual grounding score. To incorporate exploration in a unified manner, PEPO employs a smooth gating mechanism that fuses token-level entropy from the logits with the perception prior to produce a normalized token weight. These weights refine the sequence-level advantage into token-level advantages, thereby reweighting the policy-gradient updates toward visual-grounded and exploratory reasoning tokens. Moreover, PEPO integrates seamlessly with GRPO (PEPO G ) and DAPO [ 79 ] (PEPO D ), providing fine-grained optimization signals with only marginal computational overhead. To validate the effectiveness of PEPO, we evaluate it across multiple multimodal reasoning benchmarks, including geometry and math/logic reasoning, visual puzzles, visual grounding, and few-shot classification. Across Geometry3K [ 37 ] , MathVista-mini [ 36 ] , MathVerse-mini [ 83 ] , and LogicVista [ 72 ] , PEPO improves over GRPO [ 19 , 50 ] by +3.67 points on Qwen2.5-VL-3B [ 3 ] and by +3.51 points on InternVL3-2B [ 88 ] , and over DAPO [ 79 ] by +0.45 and +5.15 points, respectively. On visual puzzles (PuzzleVQA and AlgoPuzzleVQA [ 10 ] ), PEPO yields gains of +1.65 and +1.52. For visual grounding (RefCOCO [ 78 ] and LISA-Grounding [ 27 ] ), PEPO achieves +0.86 IoU@50 improvement while avoiding the entropy-only collapse. In few-shot classification (FGVC Aircraft [ 39 ] and Flower102 [ 43 ] ), it improves accuracy by +5.32 and +1.46. Furthermore, scalability analysis on ViRL39k [ 61 ] shows that perception-exploration coupling consistent gains with larger data scales, indicating robust generalization and optimization stability across multimodal tasks. To sum up, our main contributions are threefold: • To our knowledge, this is the first work to explore the complementary roles of visual-grounded and high-entropy tokens in LVLMs, revealing how perception anchors reasoning while entropy drives exploration. • We propose PEPO, a token level policy optimization framework that derives a perception prior from hidden state similarity and incorporates entropy through a smooth gating mechanism to refine advantage estimation. • We instantiate PEPO G and PEPO D on GRPO and DAPO and obtain consistent gains across geometry, math and logic, visual puzzles, visual grounding, and few-shot classification with marginal overhead. 2 Related Work \myPara RLVR for LVLMs. Reinforcement learning with verifiable rewards [ 9 , 50 , 19 , 79 ] has become an effective approach for training LVLMs. Among RLVR methods, GRPO is widely used for its stable and critic-free design that directly leverages verifiable rewards for policy optimization. Recent research has advanced this framework along two main lines of work. Data-centric studies construct large-scale multimodal datasets and adaptive training schedules to improve generalization [ 76 , 31 , 40 , 46 , 65 , 1 ,

Read the complete text