VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following

Hyesoo Hong Minsoo Kim Wonje Jeung · 2026

Read VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following by Hyesoo Hong Minsoo Kim Wonje Jeung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15672.txt.

Introduction

Hyesoo Hong, Minsoo Kim, Wonje Jeung, Sangyeon Yoon, Dongjae Jeon, and Albert No test line tracing in vision-language models. The model must track a picked path through local links. Tasks add near rivals while cutting crossings and overlaps. Even top models often drop the true path and jump to a look-alike. The fault is local rivalry: near similar distractors drag the model off course. Bigger models help little, reasoning helps at high cost, and explicit tracing orders fail. The same fault shows in tangled cables and metro maps.

Catalog record

CREATORHyesoo Hong Minsoo Kim Wonje Jeung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74140
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15672

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following Thanks: Correspondence to: Albert No <[email protected]> . Hyesoo Hong Minsoo Kim Wonje Jeung Abstract Vision-language models (VLMs) achieve strong performance on multimodal benchmarks, but may still lack robust control over basic visual operations. We study line tracing , where a model must follow a selected visual path through successive local continuations. To isolate this ability, we design controlled tracing tasks that introduce nearby competitors while reducing semantic and topological ambiguity such as crossings and overlaps. Across these tasks, even state-of-the-art VLMs frequently lose the target path and switch to nearby alternatives, especially when those alternatives look locally similar to the target. Behavioral interventions and internal analyses indicate that these failures arise from local competition: nearby similar distractors pull the model away from the true continuation. Standard remedies do not remove this bottleneck: model-size scaling provides only limited gains, reasoning partially compensates through costly substitute strategies, and explicit tracing instructions fail to recover stable path following. Finally, tests on tangled-cable scenes and metro maps with richer visual complexity show that the same path-switching failure persists beyond our controlled settings. 1 Introduction VLMs now achieve strong performance on many multimodal benchmarks [ 23 , 24 , 38 ] , yet this progress does not imply mastery of basic visual operations. A growing body of work shows that even frontier VLMs fail on visually simple tasks that humans solve easily [ 6 , 11 , 14 , 16 , 34 ] , often because models lean on language priors, semantic context, or coarse image statistics rather than genuine visual analysis [ 13 , 20 , 25 , 36 , 40 ] . Elementary capabilities such as individuating objects, sustaining attention across space, and following continuous structure remain underdeveloped [ 5 , 8 , 9 , 30 ] . In this work, our primary focus is line tracing . This operation matters disproportionately because VLMs are increasingly deployed in settings where they must follow connected visual structure, such as following a route on a transit map [ 10 , 26 ] , tracking a deformable object through a cluttered scene [ 35 ] , or reading a flow chart [ 7 , 14 , 21 , 39 ] . The shared primitive across these tasks is deceptively simple: Given a starting point, the model must stay on the selected structure as it unfolds through space. As such applications become more common, it is important to ask whether VLMs can reliably perform this basic visual operation rather than merely succeed on high-level task outputs. Prior works frame VLM failures in path tracing as limits of nonlocal reasoning under difficult conditions such as long paths, crossings, and visual shortcuts [ 4 , 8 ] . However, by conflating these factors, these works obscure the actual failure mechanism, leaving it unclear whether errors stem from poor local perception, flawed continuation choice, or lost position tracking. We resolve this ambiguity under strictly controlled conditions to reveal a systematic and reproducible deficit: VLMs struggle to trace a selected path even when nearby alternatives are simply present, and this failure becomes more severe as those alternatives share more local visual structure with the target path. Figure 1 : Two tracing settings and the corresponding model performance. In both settings, the prompt asks the model to begin from a specified start point and output the sequence of colored dots along the corresponding path in order. Left: Example of the circuit connection task and model performance. Right: Examples of Swirl tracing at Low , Moderate , and High rotation density and mean model accuracy across these three levels. To make this case, we strip our evaluation of every source of difficulty other than tracing itself. By removing crossings, overlaps, and semantic shortcuts, we create an environment where failure points to a breakdown in the core tracing mechanism. We evaluate this capability across two controlled tasks. The first is a modified circuit-connection task based on prior smooth-visual-search work [ 4 ] , designed to isolate the root cause of reported failures. The second is our central diagnostic, Swirl dataset, which removes remaining domain-specific priors by requiring models to track a single spiral. Frontier VLMs fail systematically across both tasks. Claude Sonnet 4.5 demonstrates remarkably low accuracy on both tasks, plummeting to a mere 6.2% on the modified circuit task. On dense Swirl configurations, Gemini3-Flash drops below 60%, and GPT-5.4 falls precipitously around 20%. Crucially, these failures are not random. The dominant error is a highly predictable adjacent path jump : models begin on the target path but later diverge onto a nearby distractor. Having already isolated the pure tracing operation from external confounders, these deterministic errors strongly suggest that path preservation under local competition is a consistent vulnerability. Our central claim is built on converging evidence across behavioral evaluations, causal interventions, internal representation analysis, and standard-remedy tests. We first show that the mere presence of a locally similar nearby distractor causes an immediate collapse in accuracy, and that isolating this distractor recovers performance, establishing that it actively drives the failure. Internal analyses further reveal that the model’s representations are pulled toward the nearby distractor rather than remaining anchored to the true continuation. Standard remedies do not remove this bottleneck: model-size scaling within the Qwen3-VL family produces only limited gains, reasoning-enabled models rely on costly substitute heuristics rather than stable step-by-step visual continuation, and explicit tracing instructions fail to recover performance. Finally, generalization tests on deformable line object scenes and metro maps confirm that the same path-switching failure persists under richer visual ambiguity. Our findings thus suggest that robust visual tracing under local competition may require dedicated mechanisms for maintaining the selected path across successive continuations, beyond what current architectures provide. We hope this tracing diagnostic offers a concrete way to more directly study how future VLMs maintain connected visual structure under local competition. 2 Related Work Basic Visual Abilities in VLMs. Recent work shows that strong performance on broad multimodal benchmarks can coexist with failures on visually simple tasks. Prior studies have documented broad shortcomings in the visual abilities of multimodal models [ 6 , 11 , 14 , 34 ] , as well as failures on basic geometric judgments, visual-only perception, and core visual tasks designed to reduce linguistic shortcuts [ 8 , 19 , 30 ] . Related analyses further connect such failures to limitations in binding, correspondence, and spatial reasoning [ 2 , 5 , 9 , 18 , 40 ] . These findings motivate controlled evaluations that reduce semantic and linguistic shortcuts and instead probe basic visual abilities. Our work follows this perspective, but focuses on a narrower and more diagnostic ability: keeping track of a selected path while tracing it through nearby competing continuations. Path Following and Nonlocal Visual Reasoning. A related line of work studies path following as a form of nonlocal visual reasoning, where success depends on integrating evidence across distant image regions. Prior work identifies smooth visual search as a major weakness, including contour-following and circuit-connection tasks [ 4 ] ; related map-based reasoning tasks also test route planning over transit maps [ 10 ] . Contour following has also been studied as a long-range

Read the complete text