Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models

Yanting Miao *, Yutao Sun *, Dexin Wang, Mengyu Zhou † \dagger, Pascal Poupart, Lei Lv, Li Xu, Qi Zhao, Li Wang, Hao Li, Xiaoxi Jiang, Guanjun Jiang · 2026

Read Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models by Yanting Miao *, Yutao Sun *, Dexin Wang, Mengyu Zhou † \dagger, Pascal Poupart, Lei Lv, Li Xu, Qi Zhao, Li Wang, Hao Li, Xiaoxi Jiang, Guanjun Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12374.txt.

Introduction

Miao and ten colleagues diagnose why visual latent reasoning in multimodal models gives unstable gains. Current methods feed decoder hidden states back as latent inputs, yet those states live in a very different scale regime from the embeddings the model was trained to read. Their GAP paradigm aligns the two at three levels. A light head maps decoder outputs into input-compatible latents, auxiliary visual supervision grounds the targets, and context alignment ties them to the task. The approach aims to make intermediate visual evidence reliable rather than erratic.

Catalog record

CREATORYanting Miao *, Yutao Sun *, Dexin Wang, Mengyu Zhou † \dagger, Pascal Poupart, Lei Lv, Li Xu, Qi Zhao, Li Wang, Hao Li, Xiaoxi Jiang, Guanjun Jiang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64860
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12374

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models Yanting Miao *, Yutao Sun *, Dexin Wang, Mengyu Zhou † \dagger, Pascal Poupart, Lei Lv, Li Xu, Qi Zhao, Li Wang, Hao Li, Xiaoxi Jiang, Guanjun Jiang Abstract Visual latent reasoning lets a multimodal large language model (MLLM) create intermediate visual evidence as continuous tokens, avoiding external tools or image generators. However, existing methods usually follow an output-as-input latent paradigm and yield unstable gains. We identify evidence for a feature-space mismatch that can contribute to this instability: dominant visual-latent models build on pre-norm MLLMs and reuse decoder hidden states as predicted latent inputs, even though these states occupy a substantially different norm regime from the input embeddings the model was trained to consume ( 24 ; 12 ; 19 ) . This mismatch can make direct latent feedback unreliable. Motivated by this diagnosis, we propose GAP , a G ranular A lignment P aradigm for visual latent modeling. GAP aligns visual latent reasoning at three levels: feature-level alignment maps decoder outputs into input-compatible visual latents through a lightweight PCA-aligned latent head; context-level alignment grounds latent targets with inspectable auxiliary visual supervision; and capacity-guided alignment assigns latent supervision selectively to examples where the base MLLM struggles. On Qwen2.5-VL 7B, the resulting model achieves the best mean aggregate perception and reasoning performance among our supervised variants. Inference-time intervention probing further suggests that generated latents provide task-relevant visual signal beyond merely adding token slots. 1 Introduction Multimodal large language models (MLLMs) increasingly solve visual tasks by interleaving perception and language reasoning, yet many failures on charts, high-resolution perception, and visual math stem from missing or poorly localized visual evidence 1 ; 16 ; 18 . A natural way to obtain such evidence is to call external visual tools—cropping, search, or image generation—to create auxiliary views. Tool-augmented pipelines, however, introduce extra API calls, wall-clock latency, and system complexity, and building a unified generation-and-understanding model is a substantially heavier direction that typically requires many additional visual tokens and training resources. Tool-based approaches also face cross-task generalization difficulties, since a single visual tool is rarely adequate across diverse visual reasoning tasks 17 ; 29 . We instead ask whether an image-understanding MLLM can be minimally adapted to produce intermediate visual evidence inside its own autoregressive computation. One appealing solution is visual latent reasoning : the model emits continuous latent tokens intended to represent auxiliary visual information ( 20 ; 9 ) . This design is lightweight and keeps inference inside a single model, with no external image-generation call. However, prior latent-token methods deliver unstable empirical gains, and in some settings the latent channel contributes little after post-training 10 ; 23 ; 13 . A feature-level failure mode in pre-norm MLLMs. Modern MLLMs such as Qwen3-VL 1 , Gemma 4 5 , and InternVL3.5 21 use pre-norm transformers, in which final-layer hidden states accumulate much larger norms than the input text or vision embeddings and are not guaranteed to lie in the same empirical subspace as input vision embeddings. Most recent visual-latent models build on pre-norm MLLM backbones and reuse output hidden states as next-step latent inputs ( 20 ; 9 ) . We examine this assumption on Monet-7B 20 , a state-of-the-art latent model based on Qwen2.5-VL-7B. Despite the backbone’s native final RMSNorm, its final-layer states remain far from the input-embedding distribution: text hidden states are roughly 546\times larger than text input embeddings, and vision hidden states are roughly 8.7\times larger than vision input embeddings. Re-injecting such states as latent inputs can therefore create a feature-space mismatched feedback loop that repeatedly drives the model with hidden states from a distribution it was not trained to consume as input. Interventional evidence. The norm gap is not only descriptive. Taking Monet-7B as an output-as-input latent baseline and applying a training-free Exponential Moving Average (EMA) rescaling that matches predicted latent norms to vision input-embedding norms improves performance by +2.00 on MathVista. The intervention changes neither the training data nor the backbone; it only corrects the norm of the re-injected latent vector. This supports feature-level compatibility as a practical factor for stable latent reuse: even without changing data or backbone, correcting the re-injected latent’s magnitude already recovers measurable accuracy, and the residual gap motivates the learned alignment we introduce next. Our approach: GAP. The feature-level mismatch above is the main failure mode targeted in this work, and norm rescaling alone treats only one symptom of it. Training a useful latent channel also benefits from two practical supports: the latent targets should be interpretable enough to inspect during training and analysis, and latent supervision should not be applied indiscriminately to examples the base model already solves. We therefore introduce GAP , a G ranular A lignment P aradigm for visual latent modeling across data, feature, and model levels. At the feature level, a PCA-aligned latent head maps the backbone’s native-RMS-normalized decoder states into the principal subspace of auxiliary-image vision embeddings before re-injection; the PCA basis acts both as a subspace projector that returns generated latents to the empirical vision-embedding distribution and as a low-rank parameterization of the latent head. For inspectability, context-grounded latent supervision annotates each latent example with chain-of-thought intent, a <latent> span supervised by auxiliary-image vision embeddings, and a <parser> description that records the intended auxiliary visual signal during training and analysis. For training stability, difficulty-aware latent assignment estimates base-model accuracy by repeated sampling and assigns latent targets only when the base model fails, reducing noise from unnecessary latent targets on easy examples. Figure 1 summarizes GAP as a three-level alignment recipe spanning data-, feature-, and model-level alignment. The resulting training set contains 49K curated multimodal examples spanning visual CoT, charts, geometry, multimodal math, counting, and visual search. Figure 1 : Conceptual overview of GAP. GAP addresses visual latent learning through three alignment levels: context-grounded latent supervision at the data level, a PCA-aligned latent head at the feature level, and difficulty-aware latent supervision at the model level. We make the following contributions: 1. A granular alignment paradigm (GAP). We introduce a lightweight latent feedback recipe centered on a PCA-aligned projection that returns generated latents to the vision-embedding subspace, supported by context-grounded latent supervision for inspectability and difficulty-aware latent assignment for training stability. 2. Empirical validation of aligned visual latents. Across five perception and reasoning benchmarks, GAP improves the aggregate metrics over prior visual-latent baselines and achieves the best mean aggregate supervised variant in our experiments. Controlled ablations show that PCA alignment, curated latent supervision, and difficulty-aware assignment each contribute to the gains, while inference-time interventions suggest that generated latents carry task-relevant visual signal beyond merely adding token slots. 3. Open latent-supervision data. We construct a 49K high-quality multimodal latent-supervision dataset spanning visual CoT, charts, geometry, multimodal math, counting, and visual search. We will release it to support future work on visual latent reasoning. Beyond the high-level overview in Figure 1 , Appendix Figure 3 traces the token-level execution path, showing how ordinary text decoding and GAP latent reconstruction interleave during generation. 2 Related Work Explicit visual tools and generated visual thoughts. One way to supply missing visual evidence is to call external visual operations such as zooming, cropping, or region search. ZoomEye (

Read the complete text