What’s Holding Back Latent Visual Reasoning? André G. Viveiros, Nuno Gonçalves, André F. T. Martins, Matthias Lindemann, [0.5em] Instituto Superior Técnico, Universidade de Lisboa Abstract Humans can approach complex visual problems by mentally simulating intermediate visual steps, rather than reasoning through language alone. Inspired by this, several works on Vision-Language Models have recently explored chain-of-thought reasoning with continuous latent tokens as intermediate visual “imagination” steps. In this work, we investigate how recent models leverage such latent tokens. Surprisingly, we find that model accuracy is unaffected when latent tokens are replaced by uninformative “dummy” tokens. This indicates that latent tokens play a minimal causal role in the model’s final prediction. To better understand this phenomenon, we analyze both the training signal provided by oracle latent representations and the quality of the latent tokens generated at inference time. Our experiments reveal two crucial issues holding back latent visual reasoning: First, in most existing datasets, oracle latent tokens provide limited additional information beyond the original image and do not substantially simplify the task, leading models to ignore them during training and effectively bypassing them at inference time. When fine-tuned on a diagnostic dataset, in which latent tokens provide sufficient support for the final prediction, we show that models can causally rely on them. Second, the latent tokens produced at inference time deviate from their corresponding oracle representations, collapsing to a narrow region and preventing benefits even when the model relies on them. Overall, our findings suggest that future progress in latent visual reasoning depends on two key pillars: high-quality datasets with informative intermediate steps and more precise latent token prediction. The code, models, and datasets are publicly available at LanteRn . 1 Introduction Vision-Language Models (VLMs) have achieved strong performance across a wide range of visual tasks [ 1 , 19 , 16 ] . However, they continue to struggle with scenarios that require spatial and compositional reasoning where solving a problem depends on the internal construction and manipulation of visual representations rather than purely textual descriptions [ 15 , 23 , 6 ] . In such cases, relying solely on text-based decoding can limit the model’s ability to process complex visual relationships. Latent visual reasoning has recently been explored as a pathway to address this limitation [ 18 , 5 ] . These approaches introduce continuous latent tokens as a dedicated space for visual chain-of-thought (CoT) reasoning, allowing models to move beyond text-based reasoning to manipulate visual representations, analogous to how humans approach complex visual problems [ 14 , 10 ] . Similarly to textual CoT, visual CoT decomposes a complex problem into simpler steps, but the steps operate on visual representations, such as focusing on a specific part of an image or transforming objects (see Figure 1 ). In principle, such latent tokens could enable models to visually “imagine” task-relevant transformations and provide a way for more structured and spatially aware reasoning. Figure 1: The usefulness of latent tokens depends on their content. a) When they encode a subregion of the input, it provides little auxiliary signal, resulting in the model ignoring them. b) When they capture task-relevant non-trivial processing of the input (e.g., spatial relations), this provides a stronger incentive for the model to integrate latent tokens into its reasoning to improve its answer. Prior work has reported performance improvements by using latent visual reasoning, suggesting that it can improve model capabilities [ 11 , 18 , 17 ] . Yet, it remains unclear what these latent representations actually encode, and to what extent they causally influence model predictions. In this work, we investigate the role of latent tokens in visual reasoning tasks by evaluating their causal impact on the model’s answer by replacing latent tokens with severe interventions. We find that these interventions have little to no effect on model accuracy, and in some cases, performance even slightly improves when latent tokens are completely removed. In addition, accuracy barely changes when models are conditioned on oracle latents at inference time, despite these latents containing task-relevant intermediate visual information. We see these observations as symptoms of two failure modes in current VLM training setups, a latent bypass problem and latent representation collapse . The latent bypass problem refers to the tendency of the model to ignore latent representations at inference time. We show that this issue arises from the data used in current VLM training setups (see Figure 1 a), where latent tokens rarely provide additional information that is helpful for predicting the correct answer, reducing the incentive for the model to integrate them into its reasoning process. Through fine-tuning experiments on a diagnostic dataset involving analogical reasoning over Tetris-like shape rotations, we verify this hypothesis and observe that models do rely on latent tokens when they provide information that would otherwise not be readily available in the original input. The latent representation collapse refers to the tendency of generated latents to converge toward highly similar and weakly informative representations, limiting their usefulness for reasoning at inference time. Across several models, we observe that generated latent representations exhibit extremely high similarity across samples but comparatively lower similarity to corresponding ground-truth latents, suggesting that the models fail to construct diverse and discriminative visual abstractions. Our main contributions are as follows: 1. We show that four recent off-the-shelf latent visual reasoning models [ 11 , 18 , 5 , 17 ] largely ignore the latent tokens in producing their answers ( latent bypass problem ). 2. We identify a major factor for the latent bypass problem: training data with latent tokens that only provide easily extractable information, such as image subregions. In the context of rotating Tetris-like shapes, we show that a model can learn to rely on latent tokens if they encode a non-trivial transformation of the input image. 3. Finally, we show that current methods predict latents that collapse to a narrow region of the latent space but tend to be relatively far from the ground truth ( latent representation collapse ). 2 Background: Visual Reasoning in Latent Space Latent visual reasoning methods differ in how latent representations are derived and supervised during training. To provide the context needed for our analyses, we outline the training recipes of four recent models that cover representative design choices in latent supervision: LVR [ 11 ] , Monet [ 18 ] , ILVR [ 5 ] and LanteRn [ 17 ] . At a high level, these models augment standard autoregressive decoding with latent reasoning segments, delimited by special tokens such as latent_start ( L_{s} ) and latent_end ( L_{e} ) as illustrated in Figure 1 . Upon generating an L_{s} token, the model transitions from generating discrete tokens to producing continuous latent representations, using its hidden states without projecting them through the language modeling head. These latent states are generated autoregressively, until a latent block of fixed length is complete (e.g., 6 to 12 steps, a tiny fraction of the token budget consumed by visual inputs). Once the final step is reached, an L_{e} token is automatically inserted, and the model resumes standard text generation conditioned on both the previous text inputs and latent representations. During training, the model is optimized using a combination of the standard cross-entropy objective for next-token prediction and a latent alignment objective that encourages the generated latent sequence \hat{\bm{z}} to remain close to the oracle latent sequence \bm{z}^{*} . The overall training objective is given by \displaystyle\mathcal{L}=\underbrace{-\frac{1}{T}\sum_{t=1}^{T}\log P_{\theta}(y_{t}\mid y_{