Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin · 2026
Read Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning by Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14054.txt.
Introduction
Wang and six co-authors ask whether multimodal reasoning failures come from bad seeing or bad thinking. Vision-language models chase perception-reasoning synergy, yet static text reasoning and heavy agentic scaffolds often yield a seesaw effect, lifting one skill while denting the other. The paper blames fuzzy credit assignment across modalities: when a model fails, it is unclear whether sight or logic broke. A reinforcement learning framework that rewards perception aims to settle that credit question.
Catalog record
| CREATOR | Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51977 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14054
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin Abstract Abstract Achieving robust perception-reasoning synergy is a central goal for advanced Vision-Language Models (VLMs). Recent advancements have pursued this goal via architectural designs or agentic workflows. However, these approaches are often limited by static textual reasoning or complicated by the significant compute and engineering burden of external agentic complexity. Worse, this heavy investment does not yield proportional gains, often witnessing a "seesaw effect" on perception and reasoning. This motivates a fundamental rethinking of the true bottleneck. In this paper, we argue that the root cause of this trade-off is an ambiguity in modality credit assignment: when a VLM fails, is it due to flawed perception ("bad seeing") or flawed logic ("bad thinking")? To resolve this, we introduce a reinforcement learning framework that improves perception-reasoning synergy by reliably rewarding the perception fidelity. We explicitly decompose the generation process into interleaved perception and reasoning steps. This decoupling enables targeted supervision on perception. Crucially, we introduce Perception Verification (PV), leveraging a "blindfolded reasoning" proxy to reward perceptual fidelity independently of reasoning outcomes. Furthermore, to scale training across free-form VL tasks, we propose Structured Verbal Verification, which replaces high-variance LLM judging with structured algorithmic execution. These techniques are integrated into a Modality-Aware Credit Assignment (MoCA) mechanism, which routes rewards to the specific source of error – either bad seeing or bad thinking – enabling a single VLM to achieve simultaneous performance gains across a wide task spectrum. 1 Introduction Humans navigate the world through a seamless, synergistic interplay between visual perception (seeing) and cognitive reasoning (thinking). We do not “see” once and then “think” in isolation; rather, we continuously re-interrogate visual evidence to form, verify, and refine hypotheses. This tight loop, where cognition guides perception and perception grounds cognition – a dynamic we term perception-reasoning synergy – is the hallmark of robust, generalizable intelligence. Achieving this synergy has been the central pursuit of advanced Vision-Language Models (VLMs), fueling distinct architectural paradigms. Initial architectures (e.g., Qwen-VL) focused on implicitly fusing visual tokens and textual embeddings, relying on the model’s static textual reasoning to process this fused information ( 14 ; 39 ; 34 ; 2 ) . More recent works, aspiring toward “thinking with images,” have embraced active perception by incorporating visual operations ( 31 ) , more closely reflecting the human synergistic loop. These active-perception models, however, are often implemented through complex, multi-turn function-calling or agentic frameworks. This approach introduces both significant compute and engineering burden, such as implementing multi-turn RL training and handling asynchrony for long-tailed episodes, and often yields a "seesaw effect," where marginal visual gains come at the expense of reasoning capabilities ( 10 ; 31 ) . We posit that a crucial yet under-explored bottleneck, unique to vision-language reasoning, is the lack of direct supervision signals over perception. Current training paradigms fail to provide modality-aware feedback for vision-language reasoning. Was the reasoning error caused by flawed perception (bad seeing) or flawed logic (bad thinking)? This ambiguity in modality credit assignment is overlooked and challenging because a VLM’s perception is inherently difficult to access: it is either (a) buried in latent-state activations or (b) inextricably entangled with the reasoning components in the final textual output. This entanglement makes it nearly impossible to supervise perception independently of logic. This prompts us a question: Can we improve multimodal reasoning by externalizing perception and directly rewarding it? Our answer is yes. In this work, we explore a simple way to externalize perception, leveraging instruction-following capabilities to enforce a structural separation in the generation process: perception-centric blocks (e.g., <recognition> ) that isolate visual evidence are interleaved with reasoning-centric blocks (e.g., <think> ) (Fig. 2 ). This choice transforms the opaque "black box" of VLM perception into a transparent sequence, exposing the specific locus of error – bad seeing or bad thinking – and allowing us to reframe the intractable problem of joint supervision into two solvable, component-level challenges: 1. The Perception Verification: How do we specifically supervise the quality of the model’s perception, independent of the final reasoning step? 2. The Outcome Verification: To achieve improvement across a wide task spectrum, How do we reliably reward free-form answers, avoiding the brittleness of regex and the high variance of standard LLM judges? Given isolated perception explicitly grounded in the textual space, we tackle the first challenge: the lack of ground truth for intermediate visual descriptions. We introduce Perception Verification via Proxy . Our key insight is that in explicit vision-language reasoning, visual details serve as discrete premises for logical deduction; therefore, perception sufficiency can be measured by reasoning feasibility. We implement this as a "blindfolded reasoning" test: we feed the VLM’s grounded visual evidence to a strong, text-only reasoner while withholding the image. If this text-only proxy can correctly solve the user’s question using only the VLM’s descriptions, it demonstrates that the perception block has successfully captured the sufficient statistics required for the task. This signal rewards the VLM not for generic captioning, but for extracting the precise visual facts necessary to support downstream reasoning. While the proxy validates perception, we must also ensure robust supervision for the final reasoning outcomes. Existing evaluators suffer a dichotomy: flexible LLM judges exhibit high variance, while rigid rule-based verifiers are brittle. We bridge this gap with Structured Verbal Verification (SVV) . Rather than asking an LLM for a holistic and subjective "judgment," we provide a structured natural language algorithm—a verification protocol—and instruct the judge to explicitly "execute" this protocol step-by-step. This shifts the verifier’s role from subjective estimation to structured execution, significantly reducing variance and ensuring that our reward signal remains reliable even for complex, free-form responses. we unify these techniques into a Modality-aware Credit Assignment (MoCA) mechanism. By routing these granular rewards to their specific source—punishing "bad seeing" or "bad thinking" distinctly—we dismantle the traditional "seesaw effect" where gains in one modality come at the expense of the other. Conceptually, our approach mirrors the agentic "thinking with images" paradigm but internalizes it. Rather than invoking external tools via slow, multi-turn function loops, our model treats its own perception capabilities as an internal function, calling upon it to retrieve visual evidence before reasoning. This offers a lightweight, scalable path to perception-reasoning synergy that is orthogonal to, yet distinct from, existing external agentic frameworks. Figure 1 : Overview of MoCA. MoCA identifies "bad seeing" from "bad thinking" by separating Perception and Reasoning. We introduce Perception Verification (PV, top loop), which uses a "blindfolded" text reasoner proxy to reward the VLM’s perception independently of its reasoning. This, combined with Outcome Verification (OV, bottom loop), rewards for better perception-reasoning synergy. To render an economical implementation, we use the same verifier for PV and OV. We conduct extensive empirical validation across a comprehensive suite of multimodal benchmarks. Our results demonstrate that the proposed approach outperforms state-of-the-art vision-language reasoners across a wide task spectrum spanning from perception-intensive reasoning to rich-modality reasoning. Contributions. We fundamentally rethinks VLM training by shifting from holistic outcome supervision to decoupled Modality-aware Credit Assignment (MoCA). • We enforce an explicit architectural decoupling of perception and reasoning, transforming the VLM from an opaque box into an