Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines

Keon Kim, Krish Chelikavada · 2026

Read Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines by Keon Kim, Krish Chelikavada (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15376.txt.

Introduction

Kim and Chelikavada turn a discarded byproduct of multi-step zoom-in grounding pipelines into a free confidence signal. Zoom consistency, the gap between a model's second-step guess and the crop middle, provably estimates step-1 spatial error under ideal conditions and tracks correctness across two vision-language models. Routing between specialist and generalist models on this signal captures 16.5 percent of the available oracle headroom.

Catalog record

CREATORKeon Kim, Krish Chelikavada
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21601
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15376

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines Keon Kim, Krish Chelikavada Abstract Multi-step zoom-in pipelines are widely used for GUI grounding, yet the intermediate predictions they produce are typically discarded after coordinate remapping. We observe that these intermediate outputs contain a useful confidence signal for free: zoom consistency , the distance between a model’s step-2 prediction and the crop center. Unlike log-probabilities or token-level uncertainty, zoom consistency is a geometric quantity in a shared coordinate space, making it directly comparable across architecturally different VLMs without calibration. We prove this quantity is a linear estimator of step-1 spatial error under idealized conditions (perfect step-2, target within crop) and show it correlates with prediction correctness across two VLMs (AUC = 0.60 =0.60 ; Spearman ρ = − 0.14 \rho=-0.14 , p < 10 − 6 p<10^{-6} for KV-Ground-8B; ρ = − 0.11 \rho=-0.11 , p = 0.0003 p=0.0003 for Qwen3.5-27B). The correlation is small but consistent across models, application categories, and operating systems. As a proof-of-concept, we use zoom consistency to route between a specialist and generalist model, capturing 16.5% of the oracle headroom between them (+0.8%, McNemar p = 0.19 p=0.19 ). Code is available at https://github.com/omxyz/zoom-consistency-routing . 1 Introduction Multi-step zoom-in pipelines are widely used for high-resolution GUI grounding Jiang et al. 2025 ; Luo et al. 2025 ; Pei et al. 2026 . In a typical 2-step pipeline, a VLM predicts an initial click location on the full screenshot, a crop is extracted around that prediction and resized, and the VLM re-predicts on the zoomed view. The step-2 prediction is remapped to original coordinates, and the step-1 prediction is discarded. We observe that the relationship between the step-2 prediction and the crop center encodes useful information about prediction quality. If step-1 was accurate, the target element is near the crop center, so step-2 should predict near the center. If step-1 was off, the target is displaced from the center, and step-2 must predict away from center to compensate. This displacement, which we call zoom consistency , is a free confidence signal: it requires no extra computation, no training, and no access to model internals. This paper formalizes zoom consistency, characterizes its properties, and validates it empirically. Our contributions: 1. We define zoom consistency and prove it is a linear estimator of step-1 spatial error under idealized conditions (Proposition 1 ). We characterize three failure modes where the linear relationship breaks. 2. We show that zoom consistency is directly comparable across architecturally different VLMs without calibration (Proposition 2 ), a property that log-probabilities and token-level uncertainty scores lack. 3. We validate the signal on 1,581 samples from ScreenSpot-Pro across two models (KV-Ground-8B and Qwen3.5-27B), showing significant negative correlation with accuracy ( p<10^{-6} ). 4. As a proof-of-concept application, we use zoom consistency for cross-model routing, deriving the exact condition under which such routing improves over the stronger model alone (Theorem 1 ). 2 Related Work Zoom-In Pipelines. ZoomClick Jiang et al. 2025 systematically evaluated zoom refinement for GUI grounding. LASER Luo et al. 2025 introduced preference optimization for zoom region selection. ScreenSeekeR, introduced in Li et al. 2025a , uses cascaded visual search with iterative cropping. AdaZoom-GUI Pei et al. 2026 proposed adaptive zoom with instruction refinement. Adaptive Chain-of-Focus Zhang et al. 2025 dynamically decides when to zoom. MEGA-GUI Kwak et al. 2025 proposed multi-stage coarse-to-fine grounding. These works treat zoom as a refinement mechanism; none analyze the intermediate predictions as a confidence signal. Confidence Estimation for VLMs. GUI-RC Du et al. 2025 introduced region consistency via spatial voting across multiple sampled predictions, requiring K additional forward passes per sample. Training-Free Uncertainty Guidance Kim et al. 2025 uses model uncertainty for visual input selection via repeated sampling. Self-consistency Wang et al. 2023 aggregates multiple chain-of-thought samples, designed for discrete answers rather than continuous coordinates. Adaptive VLM Routing Liu et al. 2026 routes between VLMs using a trained difficulty estimator learned on labeled data. These methods either require extra forward passes, access to model internals, or training. Zoom consistency requires none of these. GUI Grounding Models. SeeClick Cheng et al. 2024 established early GUI grounding benchmarks. OS-Atlas Wu et al. 2024 and UGround Gou et al. 2025 provided large-scale training data. GUI-Actor Wu et al. 2025 introduced coordinate-free grounding. UI-TARS Qin et al. 2025 scaled to 72B parameters. GUI-GRPO Yuan et al. 2025 applied reinforcement learning for grounding. KV-Ground-8B vocaela 2025 achieved state-of-the-art results through multi-stage fine-tuning from GUI-Owl-1.5 mPLUG 2026 . 3 Zoom Consistency 3.1 Background: 2-Step Zoom-In Pipeline Given a screenshot I of size W\times H and instruction q , a VLM predicts \hat{p}_{1}=(x_{1},y_{1}) in a normalized 1000\times 1000 coordinate space. A crop of side r\cdot 1000 centered at \hat{p}_{1} is extracted and resized back to 1000\times 1000 , and the VLM predicts \hat{p}_{2}=(x_{2},y_{2}) in the crop’s coordinate space. The coordinate mapping from original to crop space is \phi(\mathbf{q})=(\mathbf{q}-\hat{p}_{1})/r+\mathbf{m} , where \mathbf{m}=(500,500) is the crop center. The final prediction is obtained by inverting this mapping on \hat{p}_{2} . 3.2 Definition and Properties We define zoom consistency as: c=\|\hat{p}_{2}-\mathbf{m}\|_{2}=\sqrt{(x_{2}-500)^{2}+(y_{2}-500)^{2}} (1) The quantity c measures how far the step-2 prediction deviates from the center of the zoomed crop. Intuitively, if the step-1 prediction was perfect, the target sits at the crop center, so step-2 should also predict the center ( c\approx 0 ). Proposition 1 (Linear Error Estimator) . Let \mathbf{t} be the true target and \hat{p}_{1}=\mathbf{t}+\boldsymbol{\varepsilon}_{1} be the step-1 prediction with error \boldsymbol{\varepsilon}_{1} . If the target lies within the crop and step-2 perfectly identifies the target ( \hat{p}_{2}=\phi(\mathbf{t}) ), then: c=\frac{\|\boldsymbol{\varepsilon}_{1}\|_{2}}{r} (2) Proof. Under perfect step-2, \hat{p}_{2}=\phi(\mathbf{t})=(\mathbf{t}-\hat{p}_{1})/r+\mathbf{m}=-\boldsymbol{\varepsilon}_{1}/r+\mathbf{m} . Therefore \hat{p}_{2}-\mathbf{m}=-\boldsymbol{\varepsilon}_{1}/r and c=\|\boldsymbol{\varepsilon}_{1}\|/r . ∎ For a crop ratio r=0.5 , the zoom magnifies step-1 error by 2\times : a consistency of c=100 corresponds to step-1 error of 50 pixels (5% of image width). The linear relationship provides a direct, interpretable readout of localization quality. Note that the step-1-to-final coordinate shift \|\hat{p}_{1}-\text{final}\|=r\cdot c is a linear rescaling of zoom consistency, so the two signals rank identically; c is the natural form because it does not depend on r . Failure modes. The linear relationship breaks in three cases: 1. Target outside crop: When \|\boldsymbol{\varepsilon}_{1}\|_{\infty}>r\cdot 500 , the target is not visible in the zoomed view, so step-2 cannot correct toward it. The consistency value c becomes uninformative. 2. Step-2 error: If step-2 has its own prediction error \boldsymbol{\varepsilon}_{2} , the observed consistency becomes c=\|{-}\boldsymbol{\varepsilon}_{1}/r+\boldsymbol{\varepsilon}_{2}\| , bounded by |c-\|\boldsymbol{\varepsilon}_{1}\|/r|\leq\|\boldsymbol{\varepsilon}_{2}\| . This adds noise proportional to step-2’s own error. 3. Boundary clipping: When the crop extends beyond image boundaries, it is clipped, shifting the effective center. The mapping \phi no longer holds exactly. 3.3 Cross-Model Comparability Proposition 2 (Calibration-Free Comparison) . Let M_{A} and M_{B} be VLMs with different tokenizers \mathcal{T}_{A},\mathcal{T}_{B} . Their zoom consistency values c_{A},c_{B}\in[0,500\sqrt{2}] are directly comparable. In contrast, log-probability scores \ell_{A},\ell_{B} depend on token count ( L_{A}\neq L_{B} for identical coordinate strings), softmax temperature, and vocabulary size, requiring per-model calibration functions g_{i}:\mathbb{R}\to[0,1] learned on held-out data. This property arises because zoom consistency is a geometric quantity in the shared 1000\times 1000 coordinate space that both models output into. The comparison c_{A}<c_{B} is a statement about Euclidean distances, independent of how each model internally represents

Read the complete text