On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training
Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang, Nanyun Peng · 2026
Read On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training by Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang, Nanyun Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29496.txt.
Introduction
Xueqing Wu and three colleagues diagnose why post-training lifts reasoning more than perception in vision-language models, a gap that bottlenecks reasoning. Using two tasks that separate seeing from reasoning, they trace the supervised fine-tuning gap to token imbalance. Perception fills a smaller share of training text, while the reinforcement gap comes from rewards tracking reasoning more than perception. Reweighting that loss and adding a perception-aware reward raise end-to-end scores. The reweighting gain reaches 18.2 points, and the fixes transfer to benchmarks.
Catalog record
| CREATOR | Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang, Nanyun Peng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45839 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29496
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training Xueqing Wu, Yu-Chi Lin, Kai-Wei Chang, Nanyun Peng Abstract Post-training has greatly improved reasoning in frontier vision-language models, yet its gains for perception remain comparatively limited, creating a bottleneck for end-to-end visual reasoning. To investigate this gap, we introduce a controlled diagnostic framework with two synthetic tasks that disentangle perception from reasoning. Our analysis reveals a consistent perception-reasoning asymmetry : post-training improves reasoning more substantially than perception, though the underlying mechanism differs by training paradigm. For supervised fine-tuning (SFT), this asymmetry stems from token imbalance in chain-of-thought supervision, where perception occupies fewer tokens and thus receives a weaker training signal. Dynamically reweighting the loss mitigates this imbalance and boosts end-to-end performance by up to 18.2 18.2 . For reinforcement learning (RL), the asymmetry instead arises from reward coupling : outcome rewards correlate more strongly with reasoning than with perception, weakening the signal for perception learning. Adding a perception-aware reward alleviates the imbalance and improves end-to-end accuracy by up to 6.0 6.0 ; even without ground-truth perception rewards, a reliable surrogate reward provide useful signal, yielding gains of 3.2 3.2 points. Together, our results comprehensively diagnose asymmetric optimization and suggest concrete interventions to balance perception and reasoning. Figure 1: Asymmetric optimization of perception and reasoning in VLM post-training. Both SFT (left) and RL (right) improve reasoning ( R ) substantially more than perception ( P ), but with different causes and mitigation mechanism. 1 Introduction Post-training methods, especially supervised fine-tuning (SFT) and reinforcement learning (RL), have become central to modern vision-language models (VLMs). Frontier VLMs trained with these techniques exhibit increasingly strong reasoning capabilities ( 6 ; 31 ; 1 ) , achieving state-of-the-art performance across a wide range of benchmarks ( 23 ; 44 ; 45 ) . However, stronger reasoning does not necessarily imply more reliable perception: recent studies suggest that post-training may yield limited gains, or even degradations, in perception ( 21 ; 30 ; 42 ) . These observations raise a central question: why does post-training disproportionately favor reasoning over perception? Understanding the cause of this imbalance is critical, as perception errors can severely bottleneck end-to-end visual reasoning performance. To systematically study this question, we experiment in a set of controlled settings for visual reasoning. Our design is guided by two desiderata: (i) fully controllable visual inputs with ground-truth perception annotations; and (ii) a structured output format that cleanly separates perception and reasoning tokens. This design enables two complementary analyses: 1) direct evaluation of model-predicted perception, and 2) counterfactual evaluation of reasoning, where the model performs reasoning conditioned on ground-truth perception. We instantiate this framework with two synthetic tasks, graph coloring and Sudoku, using programmatically rendered images. Within this framework, we conduct controlled experiments with two strong open-source VLM families, Qwen3-VL ( 1 ) and InternVL3.5 ( 34 ) , on graph coloring and Sudoku. We identify a consistent asymmetric learning behavior , where both SFT and RL improve reasoning substantially more than perception. As a result, perception remains the dominant bottleneck for end-to-end visual reasoning, even after post-training improves the model’s ability to reason over correctly perceived inputs. To explain this asymmetry, we investigate two mechanisms: token imbalance , where perception receives little optimization signal because it occupies a smaller fraction of CoT tokens; and reward coupling , where RL outcome rewards align more strongly with reasoning than with perception. Empirical results in SFT identify token imbalance as the primary driver: perception accounts for only 2.5\% of tokens in CoT supervision, contributing just 1.3\% of the loss and 8.5\% of the gradient norm when training Qwen3-VL on graph coloring. A targeted loss-reweighting intervention confirms this mechanism. Increasing the perception weight improves perception accuracy but exposes a smooth perception-reasoning trade-off : under a fixed training budget, improving perception can weaken reasoning. Crucially, end-to-end accuracy peaks when perception is modestly upweighted , improving over standard SFT by up to 13.8 points. To avoid manual tuning, we further apply dynamic multi-task balancing methods and improve over standard SFT by up to 18.2 points. RL results, instead, suggest reward coupling as the primary driver: outcome rewards correlate strongly with reasoning correctness ( r=0.65 – 1.00 ), but much more weakly with perception correctness ( r=0.34 – 0.43 ). Augmenting outcome reward with perception reward confirms this mechanism and again exposes a perception–reasoning trade-off : increasing the perception-reward weight improves perception but weakens reasoning. In most settings, end-to-end performance peaks at a non-zero perception-reward weight , improving over standard RL by up to 6.0 points. Since ground-truth perception rewards are often unavailable, we further evaluate surrogate reward designs from prior work ( 4 ; 19 ; 39 ) . We find that reward-perception correlation serves as an effective diagnostic for predicting perception gains, with the strongest surrogate yielding a 3.2 -point improvement. Overall, our contributions are threefold: (1) We introduce a controlled framework for disentangling perception and reasoning in VLM post-training; (2) We identify a consistent perception-reasoning asymmetry and trace it to distinct mechanisms: token imbalance in SFT and reward coupling in RL; (3) We show that targeted mitigations—loss reweighting for SFT and perception rewards for RL—reduce this asymmetry and improve end-to-end performance by up to 18.2 points. 2 Analysis Framework Figure 2: Task illustration. We show the graph coloring ( GC ) and Sudoku tasks with their input images, ground truth perception, and expected reasoning respectively. 2.1 Problem Setup Let \pi_{\theta} denote a vision-language model parameterized by \theta . Given an input sequence \mathbf{x} consisting of both visual and textual tokens, the model generates the output sequence \mathbf{y}=[y_{1},\cdots,y_{T}] in an auto-regressive manner, y_{t}\sim\pi_{\theta}(\cdot\mid\mathbf{x},\mathbf{y}_{<t}) . Disentangled outputs. Standard VLM CoT entangles perception with reasoning, making it difficult to supervise them separately or attribute errors. In practice, however, CoT traces often exhibit a natural order: models first verbalize the visual information and then reason over it. Recent work starts to explicitly enforce perception-before-reasoning format; this by itself improves final performance, and further enables targeted supervision or rewards on the perception components ( 19 ; 38 ; 39 ; 4 ; 37 ) . Following this line of work, we structure each output \mathbf{y} as two textual sequences, perception \mathbf{p} followed by reasoning \mathbf{r} , separated by special tokens. Slightly abusing notation, we write the generation process as \mathbf{p}\sim\pi_{\theta}(\cdot\mid\mathbf{x}),~\mathbf{r}\sim\pi_{\theta}(\cdot\mid\mathbf{x},\mathbf{p}) . Disentangled evaluation. Our evaluation measures three metrics: end-to-end accuracy a , perception accuracy a_{p} , and reasoning accuracy a_{r} . To support this evaluation, we assume the existence of a canonical textual representation \mathbf{p}^{*} that preserves the complete visual information needed for the task. This allows us to directly evaluate perception accuracy by comparing the model-generated perception \mathbf{p} against the oracle perception: a_{p}=\mathbf{1}(\mathbf{p}=\mathbf{p}^{*}) . We further define a task-specific evaluator Acc(\mathbf{r}\mid\cdot) that checks a reasoning output \mathbf{r} over a textual perception representation rather than the raw image \mathbf{x} . The end-to-end accuracy is thus given by a=Acc(\mathbf{r}\mid\mathbf{p}^{*}) . Evaluating reasoning accuracy is nontrivial because reasoning is conditioned on perception, so reasoning errors are naturally entangled with perception errors. An intuitive metric is conditional reasoning accuracy , \tilde{a}_{r}=Acc(\mathbf{r}\mid\mathbf{p}) , which evaluates whether the reasoning is correct under the model’s own predicted perception. However, since \mathbf{p} is model-generated, this metric does not isolate intrinsic