Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection

Senjie Jin, Peixin Wang, Boyang Liu, Xiaoran Fan, Shuo Li, Zhiheng Xi, Jiazheng Zhang, Yuhao Zhou, Tao Gui, Qi Zhang, Xuanjing Huang · 2026

Read Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection by Senjie Jin, Peixin Wang, Boyang Liu, Xiaoran Fan, Shuo Li, Zhiheng Xi, Jiazheng Zhang, Yuhao Zhou, Tao Gui, Qi Zhang, Xuanjing Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03937.txt.

Introduction

Jin and co-authors question a common trick in training visual reasoning models. In text-only training, token entropy helps assign credit for good or bad steps. In image tasks, key visual tokens can have low entropy by nature. An entropy-only rule can thus skip the tokens that carry sight. Past multimodal methods felt this strain but lacked a shared fix.

Their VEPO method joins visual sensitivity with token entropy in one score. Credit flows to tokens that are both grounded in the image and rich in signal. Tests show clear gains over the entropy-only base. The gain was 2.28 points at the 7B scale and 3.15 at 3B. Extra tests back the design choices, the authors report.

Catalog record

CREATORSenjie Jin, Peixin Wang, Boyang Liu, Xiaoran Fan, Shuo Li, Zhiheng Xi, Jiazheng Zhang, Yuhao Zhou, Tao Gui, Qi Zhang, Xuanjing Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection Senjie Jin, Peixin Wang, Boyang Liu, Xiaoran Fan, Shuo Li, Zhiheng Xi, Jiazheng Zhang, Yuhao Zhou, Tao Gui, Qi Zhang, Xuanjing Huang Abstract While token-level entropy is commonly recognized as effective for credit assignment in text-only reinforcement learning with verifiable rewards (RLVR), it remains unclear whether this mechanism still holds in visual reasoning. Our controlled study shows that this mechanism collapses in visual reasoning due to the omission of vision-sensitive tokens with naturally low entropy. Although existing multimodal RL methods increasingly acknowledge the importance of visual perception, they struggle to satisfy the inherent demand for interleaving precise perceptual grounding with semantic reasoning, either lacking systematic visual measurements or overlooking that token entropy primarily drives semantic exploration. To address this, we introduce VEPO (Vision-Entropy token-selection for Policy Optimization), an effective RL framework explicitly integrating visual sensitivity with token entropy via a principled multiplicative coupling, where VEPO redirects gradient credit toward tokens which are simultaneously visually grounded and highly informative. Extensive experiments demonstrate VEPO’s leading performance, significantly outperforming the entropy-only baseline by 2.28 points at 7B-scale and 3.15 points at 3B-scale. Ablations further substantiate the soundness of our method 1 1 1 https://github.com/Leonnnnnn929/VEPO . (a) Entropy selection succeeds in LLMs but collapses in VLMs, omitting vision-critical tokens with low entropy. (b) Performance of different token-selection mechanisms on math/visual reasoning tasks. Figure 1: The entropy-based token selection mechanism collapses on visual reasoning: A small subset of high-entropy token training outperforms Full and substantially exceeds Random- k on math reasoning. But on visual reasoning, entropy-based selection only marginally beats Random- k and falls below Full . 1 Introduction Reinforcement learning with verifiable rewards (RLVR) ( Lambert et al. 2024 ; Shao et al. 2024 ; Yu et al. 2026 ) has rapidly emerged as a pivotal paradigm for incentivizing complex reasoning abilities in Large Language Models (LLMs). A fundamental challenge is how to effectively allocate credit across individual tokens within long reasoning trajectories ( Zhong et al. 2024 ; Wang et al. 2026 ) . Recent work in text-only RL converges on a compelling empirical regularity: a small fraction of high-entropy tokens, dubbed forking tokens , dominate exploration and supply the majority of positive gradient signals ( He et al. 2026 ; Wang et al. 2025b ) . Token-level entropy H_{t} thereby serves as a remarkably reliable proxy for credit assignment and policy improvement ( Cui et al. 2025 ; Xi et al. 2026 ; Wang et al. 2025c ) . While this entropy-driven mechanism proves highly effective for text-only reasoning tasks, whether it generalizes to visual reasoning remains a critical open question ( Li et al. 2025 ) . We investigate this through a controlled study ( Section 2.1 ) and observe a striking divergence: the entropy mechanism collapses once the visual modality is introduced (Figure 1(b) ). On math reasoning (AIME’24), we reproduce the prior findings ( Wang et al. 2026 ) , yet the identical mechanism underperforms the full-token GRPO baseline on visual reasoning tasks, even yielding marginal gains over the random selection strategy. We further probe the underlying cause and identify a previously uncharacterized phenomenon ( Section 2.2 ): the pure entropy-based token selection inadequately captures genuinely vision-sensitive tokens ( Figure 2 ) that possess naturally low entropy ( Favero et al. 2024 ; Leng et al. 2024 ) . Quantitatively, at k{=}0.2 , top-entropy selection recovers merely 60\% of tokens flagged by the vision-sensitive metrics, thereby we argue that token entropy alone remains insufficient for identifying the critical tokens in visual reasoning. Qualitative cases are deferred to Figure 2 (c)(d) and Appendix J . Unlike text-only reasoning tasks, visual reasoning intrinsically requires interleaving precise perceptual grounding with semantic reasoning ( Gu et al. 2025 ; Su et al. 2025b ) . While recent work in multimodal RL ( Liu et al. 2025a ; Wang et al. 2025d ; Li et al. 2026b ) has increasingly recognized the importance of visual perception, they either lack systematic visual signal measurement (e.g., similarity, KL Divergence) ( Li et al. 2026b ) , or neglect the fact that entropy ( Huang et al. 2025 ; Ye et al. 2026 ) primary serve as a proxy of semantic exploration, which visual modalities cannot effectively resolve. Motivated by above analysis, we propose VEPO (Vision-Entropy token-selection for Policy Optimization), an effective RL framework that explicitly couples vision-sensitivity with token entropy during token selection, redirecting gradient credit toward tokens that serve a true functional purpose in visual reasoning. Concretely, VEPO performs a counterfactual forward pass with a noise-perturbed image and quantifies token-level visual dependency by multiplicatively coupling two complementary signals: (i) the Jensen–Shannon divergence (JSD) ( Menéndez et al. 1997 ) for distributional shift, and (ii) the absolute entropy gap |\Delta H_{t}| for direction-agnostic shifts. This joint formulation is conceptually inspired by the aleatoric–epistemic uncertainty decomposition ( Houlsby et al. 2011 ; Depeweg et al. 2018 ; Kendall and Gal 2017 ) , establishing distributional divergence and predictive confidence as complementary axes. Furthermore, the multiplicative coupling acts as a soft probabilistic OR ( Xu et al. 2026 ; Pearl 2014 ) , guaranteeing the comprehensive capture of visually sensitive signals by preventing the omission of tokens that spike along either axis ( Appendix G ). Ultimately, by modulating this fused signal with token entropy, VEPO ensures policy updates are driven exclusively by visual forking tokens that are simultaneously visually grounded and highly informative. We evaluate VEPO with Qwen2.5-VL-7B/3B-Instruct ( Bai et al. 2025 ) . Experimental results demonstrate that VEPO achieves leading performance at the 7B scale, outperforming the top-entropy baseline by +2.28 , with consistent gains extending to the 3B scale. Moreover, we conduct comprehensive ablation studies and analysis ( Section 5 ) to validate the soundness and effectiveness of our proposed method. Figure 2: (a) Many high-JSD / high- |\Delta H| tokens lie in the low-entropy region. (b) Top-Entropy selection misses 41\% of Top-JSD tokens at k\!=\!20\% , alongside a comparable proportion of Top- |\Delta H| tokens. (c,d) Tokens flagged by JSD / |\Delta H| (red triangles) frequently fall in entropy valleys and are missed by Top-Entropy selection. Our contributions are summarized as follows: • We provide a comprehensive diagnosis revealing why the top-entropy token selection mechanism collapses in visual reasoning, attributing to its inherent omission of visually grounded tokens. • We propose VEPO , a novel RL framework that integrates visual sensitivity and token entropy signals to achieve a visually grounded and highly informative optimization. • Extensive experiments across seven benchmarks and two model scales, alongside comprehensive ablation studies, demonstrate the effectiveness and soundness of VEPO. 2 Preliminary Experiments Recent works on math reasoning have uncovered a striking empirical observation: training on a small subset of high-entropy tokens yields non-trivial benefits ( Wang et al. 2026 ; He et al. 2026 ; Wang et al. 2025b ) . Building upon this, we reproduce the experiments and further investigate whether this mechanism generalizes to visual reasoning tasks. Our empirical findings and analyses are elaborated in the subsequent sections, while detailed experimental setup is deferred to Appendix A . 2.1 The entropy mechanism collapses on visual reasoning The entropy-based token-selection mechanism

Read the complete text