Latent Denoising Improves Visual Alignment in Large Multimodal Models CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning
Dhruv Parikh, Jacob Fein-Ashley, Rajgopal Kannan, Viktor Prasanna · 2026
Read Latent Denoising Improves Visual Alignment in Large Multimodal Models CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning by Dhruv Parikh, Jacob Fein-Ashley, Rajgopal Kannan, Viktor Prasanna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21343.txt.
Introduction
Dhruv Parikh and three colleagues strengthen large multimodal models' visual grounding with latent denoising, countering weak visual features and brittleness under shift. Their method corrupts visual tokens with a saliency-aware mix of masking and Gaussian noise, then trains the model to recover clean teacher patch features from a middle layer. Keeping intra-image similarity structure and contrastive patch training stops collapse. The method improves visual understanding across benchmarks, holds up better under common corruptions, and adds no inference overhead.
Catalog record
| CREATOR | Dhruv Parikh, Jacob Fein-Ashley, Rajgopal Kannan, Viktor Prasanna |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76012 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21343
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Denoising Improves Visual Alignment in Large Multimodal Models CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning Dhruv Parikh, Jacob Fein-Ashley, Rajgopal Kannan, Viktor Prasanna Abstract . Large Multimodal Models (LMMs) such as LLaVA are typically trained with an autoregressive language modeling objective, providing only indirect supervision to visual tokens. This often yields weak internal visual representations and brittle behavior under distribution shift. Inspired by recent progress on latent denoising for learning high-quality visual tokenizers, we show that the same principle provides an effective form of visual supervision for improving internal visual feature alignment and multimodal understanding in LMMs. We propose a latent denoising framework that corrupts projected visual tokens using a saliency-aware mixture of masking and Gaussian noising. The LMM is trained to denoise these corrupted tokens by recovering clean teacher patch features from hidden states at a selected intermediate LLM layer using a decoder. To prevent representation collapse, our framework also preserves the teacher’s intra-image similarity structure and applies intra-image contrastive patch distillation. During inference, corruption and auxiliary heads are disabled, introducing no additional inference-time overhead. Across a broad suite of standard multimodal benchmarks, our method consistently improves visual understanding and reasoning over strong baselines, and yields clear gains on compositional robustness benchmarks (e.g., NaturalBench). Moreover, under ImageNet-C-style non-adversarial common corruptions applied to benchmark images, our method maintains higher accuracy and exhibits reduced degradation at both moderate and severe corruption levels. Our code is available at https://github.com/dhruvashp/latent-denoising-for-lmms . 1. Introduction Large Language Models (LLMs), spanning proprietary systems such as GPT 5.4, Gemini 3.1 Pro, and Claude 4.6 Opus ( 62 ; 26 ; 2 ) , as well as open-source models such as Llama 4, DeepSeek-V2, and Qwen3-Next ( 58 ; 17 ; 66 ) , have rapidly become the backbone of modern AI systems and applications. Building on this progress, Large Multimodal Models (LMMs) have emerged as a dominant paradigm for jointly reasoning over language and visual content. A large class of open-source LMMs follows a simple yet powerful recipe: pairing a pretrained LLM with a strong vision encoder through a lightweight projector and performing multimodal instruction tuning, as exemplified by LLaVA ( 49 ; 48 ; 47 ) and its successors such as Qwen-VL and InternVL ( 13 ; 12 ; 11 ; 4 ; 81 ; 6 ; 5 ) . In parallel, another line of work explores models trained natively in a multimodal fashion through large-scale joint pretraining across vision and language, including proprietary systems such as GPT-4o and Gemini ( 61 ; 73 ) as well as open-source models such as Kosmos-2, Molmo, and Transfusion ( 64 ; 14 ; 75 ) . Such LMMs have demonstrated strong capabilities across a wide range of complex visual understanding and multimodal reasoning tasks, including visual question answering and compositional visual reasoning ( 3 ; 34 ; 27 ) , document and scene text understanding ( 70 ; 57 ; 60 ) , chart and infographic reasoning ( 54 ; 56 ; 59 ) , multimodal knowledge-intensive reasoning and large-scale multimodal evaluation benchmarks ( 88 ; 50 ; 42 ) , and real-world visual reasoning under noisy or unconstrained imagery ( 7 ; 41 ; 84 ) . Despite these advances, recent studies increasingly show that modern LMMs remain imperfect visual reasoners. Under cross-modal conflict, they can place disproportionate trust in textual inputs and follow misleading language cues over grounded visual evidence ( 18 ) . They also remain prone to object hallucination and image-context reasoning failures, even when prompted on seemingly simple visual facts ( 46 ; 28 ) . Complementary analyses of their internal behavior further reveal that visual evidence is often weakly utilized: LMMs can exhibit visual attention sink effects, rapidly diminishing contribution of visual tokens with depth, and highly sparse visual grounding behavior concentrated in only a few attention heads ( 37 ; 8 ; 38 ) . More broadly, recent work on multimodal representation analysis suggests that these models continue to suffer from imperfect cross-modal alignment and incomplete modality integration within the LLM pathway ( 55 ; 33 ) . Collectively, these findings suggest that, despite strong task-level performance, current LMM training still provides insufficient direct supervision to internal visual representations, leaving them vulnerable to text-dominant behaviors and brittle multimodal reasoning. Recent work has begun to address this limitation more directly. Some approaches first strengthen the visual backbone itself, for example by post-training vision encoders for better locality or aligning CLIP-like features to stronger vision-centric representations, thereby improving the visual inputs supplied to downstream LMMs while remaining largely external to end-to-end multimodal instruction tuning ( 15 ; 25 ) . Others inject explicit supervision into the multimodal stack by guiding projector outputs or internal visual tokens using refined shallow-layer embeddings, patch-level alignment, or alignment to one or more vision foundation models ( 72 ; 36 ; 87 ; 82 ) . A third line introduces reconstructive visual supervision through latent reconstruction, masked image modeling, or diffusion-based objectives ( 80 ; 44 ; 71 ) . Collectively, these studies make clear that direct visual supervision matters (Figure 1 ); however, existing approaches remain fragmented across backbone refinement, feature alignment, and reconstructive supervision, and often rely on specialized teachers, auxiliary generators, or narrowly scoped supervision objectives. As a result, a more unified and efficient mechanism for refining internal visual representations within LMMs remains underexplored. Figure 1. Comparison of visual supervision paradigms for LMMs. (a) Standard training: language loss only; visual tokens indirectly constrained. (b) Representation alignment: features aligned to external teachers ( 72 ; 87 ; 82 ; 36 ) . (c) Reconstructive supervision: latent or diffusion-style reconstruction ( 80 ; 44 ; 71 ) . (d) Ours : saliency-guided corruption of projected visual tokens via Gaussian noising and masking, with recovery of clean teacher targets from intermediate hidden states. Motivated by this gap, we draw inspiration from a recent line of work suggesting that high-quality visual latent spaces need not be specific to either understanding or generation. On the generation side, semantically rich representation spaces have been shown to support strong diffusion-based image synthesis through representation autoencoders, while more recent unified multimodal pretraining results suggest that a shared visual representation can simultaneously benefit visual understanding and generation within a single model ( 93 ; 76 ; 75 ) . In parallel, recent work on visual tokenizers shows that latent denoising —recovering clean latent targets from corrupted latent inputs, e.g., under masking or Gaussian perturbations—can substantially improve the quality of learned visual tokens for downstream generation ( 85 ) . These developments motivate a simple question for large multimodal models: if latent denoising improves the quality of visual token spaces for generation, can it also serve as an effective form of supervision for the internal visual features used for multimodal understanding? Guided by this perspective, we propose a training-time latent denoising framework for large multimodal models (Figure 2 ). Rather than relying on target-text supervision alone to shape visual representations, our method perturbs projected image tokens using a saliency-aware mixture of masking and Gaussian noising ( 85 ) , and trains the model to recover aligned teacher patch features from these corrupted tokens. The supervision is applied to an internal LMM