LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning Linquan Wu, Tianxiang Jiang, Yifei Dong, Haoyu Yang, Fengji Zhang, Shichang Meng, Ai Xuan, Linqi Song, Jacky Keung Abstract Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher’s textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT , a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher’s visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o. † † * Equal contribution. 1 Introduction Multimodal Large Language Models (MLLMs) have advanced rapidly in recent years ( 1 ; 5 ; 34 ) , early multimodal reasoning models primarily thinking about images , captioning visual inputs into text and reasoning mainly in the language space via explicit chains of thought ( 16 ; 47 ; 28 ) . Recent work instead emphasizes thinking with images , more tightly integrating visual evidence into the reasoning process ( 15 ; 52 ; 32 ) , leading to improved performance on complex visual reasoning tasks ( 10 ; 46 ; 49 ) . Figure 1: Conceptual Illustration of Our Proposed Method LaViT. Subsequently, latent reasoning ( 13 ) has emerged as a complementary direction, compressing intermediate reasoning into continuous hidden states rather than explicit CoT. This paradigm has been extended to MLLMs to model abstract visual thoughts within latent tokens 48 ; 22 . While effective, existing methods largely rely on manually designed visual supervision, such as auxiliary images or annotated regions, leaving intrinsic visual attention dynamics during reasoning unexplored. These limitations motivate the use of knowledge distillation ( 14 ) as a lens to analyze and transfer visual reasoning behaviors in MLLMs. Distilling a high-capacity teacher into a compact student enables us to probe not only what knowledge is transferred, but also how visual reasoning is internally realized. However, existing multimodal distillation methods mainly align final textual outputs or distributions ( 3 ; 30 ) , implicitly assuming that reproducing answers or static representations suffices to inherit multimodal reasoning ability. To examine this assumption, we conduct empirical analyses, revealing a pronounced mismatch between textual alignment and visual reasoning: (I) Correct multimodal reasoning is causally constrained by focused visual attention: when models fail to attend to relevant regions, hallucinated or unreliable responses emerge. (II) Even when student models closely match teacher outputs under standard distillation, their visual attention trajectories can diverge substantially, particularly for tasks requiring fine-grained visual grounding. Together, these findings expose a fundamental Perception Gap in multimodal distillation: students often learn what to say without learning where to look , instead relying on language priors rather than grounded visual evidence. Motivated by this insight, we propose LaViT , a distillation framework that aligns latent visual thoughts rather than static visual embeddings. LaViT trains the student to autoregressively generate continuous latent tokens that reconstruct the teacher’s internal visual semantics and attention trajectories prior to textual response generation, explicitly transferring both what visual concepts to encode and where to attend during reasoning. To prevent shortcut learning through direct access to visual features, we introduce Curriculum Sensory Gating , which progressively restricts and then relaxes visual input during training. This strategy enforces a latent bottleneck early on, compelling reliance on latent visual reasoning while avoiding training–inference mismatch. Extensive experiments demonstrate that LaViT substantially improves both visual grounding and multimodal reasoning. LaViT-3B achieves up to +5.0% gains on fine-grained perception benchmarks MMVP ( 38 ) and substantial improvement on BLINK 10 , outperforming strong baselines and rivaling or surpassing 7B models and proprietary GPT-4o. 2 Related Work 2.1 Visual Chain-of-Thought Originating from text-only LLMs ( 43 ) , Chain-of-Thought (CoT) has expanded to multimodal contexts ( 27 ) . Following DeepSeek-R1 ( 12 ) , recent works enhance multi-step visual reasoning via RL-style optimization ( 16 ; 47 ; 28 ; 8 ; 18 ) ; however, these methods primarily rely on indirect textual proxies rather than intrinsic visual understanding. Conversely, a parallel stream shifts to thinking with images by orchestrating tools ( 50 ; 45 ; 31 ; 39 ; 51 ) , utilizing executable programs ( 15 ) or iterative region grounding ( 52 ) . Furthermore, unified architectures now support interleaved generation, exemplified by Chameleon’s unified tokens ( 35 ) , MVoT’s multimodal trajectories ( 23 ) , and other general-purpose frameworks ( 37 ; 6 ; 11 ) . 2.2 Latent Reasoning Recent advances have shifted the reasoning paradigm from discrete token sequences to continuous hidden states, effectively enhancing both computational efficiency and flexibility ( 13 ; 29 ; 44 ) . Extending this concept to multimodal learning, current MLLMs align specialized latent tokens with visual embeddings derived from auxiliary supervision signals, such as helper images ( 48 ) or annotated bounding boxes ( 22 ) . To further improve grounding, CoVT ( 26 ) integrates fine-grained perceptual priors from models like DINO ( 25 ) and SAM ( 21 ) , while other approaches explore interleaved patterns to mimic internal visual imagination ( 36 ; 40 ) . However, these existing methods primarily constrain latent tokens using static encoder features, critically overlooking the dynamic guidance offered by attention maps. 2.3 Knowledge Distillation Knowledge distillation ( 14 ) , which transfers capabilities from a high-capacity teacher to a compact student, has been widely adopted in LLMs via logit matching ( 33 ; 19 ) . Extending this paradigm to multimodal models, DistillVLM ( 7 ) performs transformer distillation by using an MSE loss to match the teacher and student’s hidden attention distributions and feature maps. In contrast, MAD ( 41 ) emphasizes aligning visual and textual token features between teacher and student, leveraging token selection to guide the matching. More recent research explores distillation tailored to MLLMs for specific downstream tasks, including visual grounding ( 3 ; 9 ) and compositional learning ( 20 ) . 3 Empirical Analysis of Perception Gap We conduct a pilot study to quantify the misalignment between textual generation and visual attention, centered on two research questions: (RQ1) Is correct visual reasoning causally linked to focused visual attention? (i.e., Does looking at the right place precondition the right answer? ) (RQ2) Does a significant “alignment gap” exist in the visual trajectories between teacher and student models, even when their textual outputs are similar? 3.1 Visual Attention Dictates Reasoning Bounds Definition 3.1 (Visual Focusing Score). Let I be the image and B_{gt} the target bounding box. Given the model’s aggregated attention trajectory \mathcal{A}_{traj}\in\mathbb{R}^{H\times W} , which accumulates attention weights across all layers and heads, the visual focusing score S_{focus} is defined as: S_{focus}=\frac{\sum_{(u,v)\in B_{gt}}\mathcal{A}_{traj}(u,v)}{\sum_{(u,v)\in I}\mathcal{A}_{traj}(u,v)} (1) where \mathcal{A}_{traj}(u,v) denotes the attention intensity at spatial coordinate (u,v) . The denominator represents the total attention mass distributed across the entire image I . A larger S_{focus} indicates a stronger dependency of the reasoning process on the verified visual evidence, implying that the model is actively “looking” at the semantically correct region rather than relying on language priors or blind guessing. Building upon the above metric, we analyze the attention trajectories of Qwen2.5-VL-32B 2 on 1,000 randomly sampled instances from the Visual-CoT ( 27 ) . As illustrated in Figure 2 , reasoning outcomes are strictly constrained by the intensity of visual attention: • Monotonic Performance Gain: We observe that reasoning accuracy improves monotonically as the S_{focus} threshold increases. Statistical analysis confirms that correct samples maintain a significantly higher average S_{focus} ( 15.89% ) compared to incorrect ones ( 11.84% ), indicating a substantial relative gap of \sim 34%. This validates that higher visual energy is a strong predictor of reasoning success. • Visual Absence and Hallucination: Conversely, in samples with negligible S_{focus} ( <1\% ), we predominantly observe responses that are completely irrelevant to the visual content or contain severe hallucinations. This pattern suggests that without active visual grounding, the model relies on language priors to blindly guess, which proves to be a highly unreliable strategy for complex visual tasks. Figure 2: Impact of Visual Attention on Reasoning Accuracy. The monotonic increase in accuracy with higher Visual Focusing Score ( S_{focus} ) thresholds validates that effective visual grounding is a prerequisite for correct reasoning. Observation 1: Visual attention is determinative, not merely interpretative. The strict positive correlation confirms that focused visual grounding ( S_{focus} ) is a necessary condition for reasoning success. Models cannot reason correctly without “looking” at the right evidence, effectively ruling out blind guessing as a viable strategy. 3.2 Perception Gap between Teachers and Students Given the critical role of visual attention, we investigate whether standard Supervised Fine-Tuning (SFT) enables student models to inherit the teacher’s visual thinking process. This analysis is conducted on the same 1,000 samples following the experimental setup in Section 3.1 . Analysis of Attention Divergence. We fed identical reasoning prompts to both the Teacher ( M_{T} ) and the Student ( M_{S} ). We categorized the generated tokens into three groups based on their semantic reliance on visual evidence: Functional (e.g., stop words), Object (nouns), and Attribute (adjectives, spatial relations). We then computed the Kullback-Leibler (KL) divergence between their normalized attention maps \mathcal{A}^{T} and \mathcal{A}^{S} , alongside the Cosine Distance of their hidden states. As shown in Figure 3 , we observe a decoupling between textual alignment and visual attention: • Attention Drift on Visual Concepts: The attention divergence exhibits a distinct monotonic increase as the semantic reliance on vision deepens. While Functional tokens maintain a lower average divergence ( \mu=1.11 ), Attribute tokens, which require precise visual grounding, show the highest misalignment ( \mu\approx 1.39 ). This indicates that when describing fine-grained details (e.g., color, texture), the student struggles to focus on the same regions as the teacher. • The “Blind Guessing” Phenomenon: Crucially, while the attention divergence surges, the Cosine Distance of the hidden states remains relatively stable across categories (ranging from 0.52 to 0.55 ). This implies that the student can mimic the teacher’s textual representations (learning what to say) without correctly aligning its visual attention (learning where to look), effectively relying on language priors rather than active observation. Observation 2: Textual mimicry does not guarantee visual understanding. SFT trains the student to reproduce the teacher’s words but fails to transfer the underlying visual trajectory . This “Perception Gap” suggests that the student model is often “guessing” based on language context rather than actively “observing” the image. Figure 3: The Perception-Reasoning Gap. While the student aligns closely with the teacher in textual representations (stable Cosine Distance), their visual attention trajectories diverge significantly on attribute-heavy tokens (rising KL Divergence). This reveals that textual mimicry does not imply visual grounding . 4 Method 4.1 LaViT-SFT-15K We construct LaViT-SFT-15K , comprising 15K tuples \langle I,Q,A,\mathcal{A}_{traj},\mathcal{V}_{sem}\rangle , to distill the Internal Cognitive States of the teacher (Qwen2.5-VL-32B ( 2 ) ). Unlike tool-based approaches, we directly extract intrinsic reasoning traces. Data quality is enforced via a Three-Stage Filtering pipeline: (1) Correctness: retaining samples matching ground truth; (2) Difficulty: removing instances solvable by a text-only model; and (3) Alignment: rejecting samples with <20\% aggregated attention mass falling within the target regions delineated by the ground-truth bounding box annotations in Visual-CoT ( 27 ) , thereby excluding non-visually grounded hallucinations. We extract two white-box signals. First, Dynamic Visual Gaze ( \mathcal{A}_{traj} ) represents the attention trajectory. Given text sequence T_{text} , we aggregate cross-attention weights A^{(l,h)}_{i,j} across layers L and heads H : S_{j}=\frac{1}{L\cdot H\cdot|T_{text}|}\sum_{l=1}^{L}\sum_{h=1}^{H}\sum_{i\in T_{text}}A^{(l,h)}_{i,j} (2) We further apply Min-Max normalization to obtain the final gaze probability: \mathcal{A}_{traj}(j)=\frac{S_{j}-\min(S)}{\max(S)-\min(S)+\epsilon} (3) Crucially, unlike static visual features extracted from a frozen vision encoder, our target \mathcal{V}_{sem} is derived from the Teacher’s last transformer layer. Due to the self-attention mechanism, these image token representations have effectively interacted with the textual instructions Q . Therefore, \mathcal{V}_{sem} represents contextualized visual thoughts—reflecting not just what is in the image, but how the Teacher interprets the visual content specifically in response to the given query. To distill the most salient visual cues, we subject \mathcal{A}_{traj} to Top-K ( k=8 ) sparsification, thereby ensuring sparse and noise-free supervision. 4.2 White-box Trajectory Distillation We establish the foundational reasoning capability of our model through a Supervised Fine-tuning (SFT) stage defined as Latent Teacher Forcing . Unlike standard SFT that maps inputs directly to text, we train the model to autoregressively generate a sequence of K continuous latent tokens, \mathbf{V}=\{,\dots,\} , prior to producing the textual response. These latent tokens serve as Visual Information Containers . By leveraging a white-box distillation approach, we force \mathbf{V} to explicitly capture and compress the teacher’s high-dimensional visual semantics and gaze patterns. Formally, given an input [I,X_{q}] , the model generates the latent and textual sequences to form the complete trajectory X=[I,X_{q},\mathbf{V},X_{ans}] , where \mathbf{V} acts as the indispensable cognitive bridge supplying visual evidence for the subsequent response X_{ans} . 4.2.1 Curriculum Sensory Gating A naive attention mechanism allows response tokens X_{ans} to attend directly to image patches I , enabling the model to bypass \mathbf{V} (shortcut learning). Conversely, a permanent hard mask creates a training-inference distribution shift. To resolve this, we propose Curriculum Sensory Gating, which modulates the direct visual perception path via a time-dependent scalar \gamma(t)\in[\epsilon,1] . We implement gating within the attention bias of the Transformer. Let Q_{txt} denote queries from response tokens and K_{img} denote keys from image patches. The attention scores are computed as: \begin{split}\text{Attn}(Q_{txt},K_{img})&=\text{Softmax}\left(\frac{Q_{txt}K_{img}^{\top}}{\sqrt{d}}+\mathbf{B}_{gate}(t)\right),\\ \text{s.t. }\mathbf{B}_{gate}(t)&=\ln(\gamma(t)).\end{split} (4) To ensure a structured internalization process mentioned above, we introduce a warm-up period T_{w} . The gating scalar \gamma(t) is defined as: \gamma(t)=\begin{cases}\epsilon+\frac{1-\epsilon}{2}\left[1-\cos\left(\frac{\pi t}{T_{w}}\right)\right],&t