BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs

Aaditya Baranwal, Vishal Yadav, Abhishek Rajora · 2026

Read BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs by Aaditya Baranwal, Vishal Yadav, Abhishek Rajora (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10528.txt.

Introduction

Baranwal, Yadav, and Rajora test whether vision language models truly see shape. Rich color and scene hints may let models fake shape sense with texture shortcuts. Their BareBones set strips scenes to bare silhouettes across six data pools. A flagship puzzle set, WTP-Bench, forces calls from edge lines alone. All 26 tested models crash under color loss, a drop the authors dub the Texture Bias Cliff.

Catalog record

CREATORAaditya Baranwal, Vishal Yadav, Abhishek Rajora
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38099
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10528

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs Aaditya Baranwal, Vishal Yadav, Abhishek Rajora Abstract While Vision-Language Models (VLMs) demonstrate remarkable zero-shot recognition capabilities across a diverse spectrum of multimodal tasks, it yet remains an open question whether these architectures genuinely comprehend geometric structure or merely exploit RGB textures and contextual priors as statistical shortcuts. Existing evaluations fail to isolate this mechanism, conflating semantic reasoning with texture mapping and relying on imprecise annotations that inadvertently leak environmental cues. To address this gap, we introduce BareBones , a zero-shot benchmark designed to stress-test pure geometric shape comprehension. We curate pixel-level silhouettes of geometrically distinct classes across six datasets: five established segmentation sources (ImageNet-S, DIS5K, ThinObject5K, PASCAL VOC, CUB-200) and our novel flagship collection, WTP-Bench, establishing a noise-free geometric taxonomy. WTP-Bench is an extreme, fine-grained visual puzzle that forces models to identify inter-class geometric concepts from boundary contours alone. Our evaluation of 26 state-of-the-art proprietary and open-weight VLMs ( e.g . , GPT-4.1, Gemini, Claude Sonnet 4.5, LLaVA) reveals a consistent, severe performance collapse under RGB deprivation, a phenomenon we term the Texture Bias Cliff . By documenting universal structural blindspots, BareBones establishes a rigorous yardstick for genuine geometric grounding. Figure 1 : Evaluation pipeline : HQ images and their segmentation masks are binarized to pure structural silhouettes, eliminating all color, texture, and background cues. Models are queried in two modes (HQ and Silhouette), and exact-match accuracy is recorded. 1 Introduction Recent advancements in Vision-Language Models (VLMs) have demonstrated remarkable zero-shot capabilities across visual recognition, reasoning, and multimodal generation. Through massive-scale pretraining, models such as GPT-4.1 22 , Gemini 11 , LLaVA 18 , and Qwen-VL 4 achieve impressive compositional scene understanding. Yet it remains an open question whether these models genuinely comprehend geometric structure, or whether they primarily exploit RGB textures, color histograms, and contextual priors as statistical shortcuts . A texture-reliant model is brittle by construction: it fails under novel appearances, unusual lighting, or any condition severing the surface-pattern shortcut, a critical limitation for real-world deployment. Benchmarks 28 ; 16 ; 19 ; 5 ; 24 fail to isolate this geometric gap. Standard recognition datasets conflate semantic reasoning with texture mapping: the distinctive stripes of a tiger suffice without any shape comprehension. Loose bounding-box annotations leak contextual cues, while polygon segmentations inadequately capture fine structural boundaries. Progress on these benchmarks masks fundamental visual deficits, motivating harder evals. We introduce BareBones : a zero-shot benchmark that stress-tests VLMs by removing all color, texture, and background, leaving only pixel-level silhouettes. Evaluated across six high-fidelity segmentation taxonomies and 26 frontier models (7 proprietary, 19 open-weight from 1B to 26B), BareBones exposes a severe Texture Bias Cliff : a consistent, large-scale performance collapse under RGB deprivation that is (1) universal across architectures, (2) invariant to parameter scale, and (3) rooted in hallucination of pretraining priors rather than downstream reasoning failures. Our flagship dataset, WTP-Bench (1,160 Pokémon silhouettes spanning 8 generational tiers), provides a human-tractable ( \sim 70% for dedicated fans) yet model-impenetrable ( 3.10\% best open-weight) extreme test of fine-grained geometric retrieval. 2 BareBones Benchmark Curated Taxonomy : BareBones repurposes five established segmentation datasets and introduces one novel flagship collection, spanning coarse objects to extreme fine-grained boundaries ( Table 1 ). ImageNet-S 8 provides shape-discriminative semantic classes, filtered to ensure perceptually distinct silhouettes. DIS5K 25 and ThinObject5K 17 stress-test complex pixel-perfect boundaries including intricate negative spaces and thin-wire geometries. PASCAL VOC 7 contributes articulated poses and environmental occlusions. CUB-200 29 covers 200 bird species whose silhouettes differ only in beak curvature, tail length, and crest geometry: a stringent test of fine-grained inter-class discrimination as a relevant visual challenge. WTP-Bench is our flagship fine-grained visual puzzle: 1,160 Pokémon silhouettes spanning 8 generational tiers, inspired by the classic “Who’s That Pokémon?” interlude. Species share closely related base geometries yet differ in intricate localized horns, wings, or tails, demanding precise structural boundary retrieval. The evolutionary taxonomy enables a tiered evaluation : Master Ball (exact species + form), Ultra Ball (correct species, wrong form), and Great Ball (correct evolutionary lineage), isolating whether errors reflect form-level ambiguity or deeper family-level confusion ( Table 7 ). Sprite assets are derived from the publicly available PokéSprite API; binarized masks constitute transformative, non-commercial fair use. Task non-triviality: dedicated fans achieve \sim 70% exact-match accuracy on Generation 1 challenges, a human ceiling that frontier open-weight models fall dramatically short of ( 3.10\% best), confirming near-zero model scores reflect a genuine capability gap, not task impossibility. Shape-Only Canonicalization & Evaluation : For each source image, the dominant segmentation mask is binarized to a pure structural silhouette (white foreground on black background), following standard segmentation convention. All figures display an inverted palette for visual clarity. Silhouettes are cropped to their bounding box and resized to a 200px maximum dimension, standardizing input scale across all encoders. Evaluations span two modes: HQ Reference (full-color, establishing the texture upper bound) and Silhouette (binarized mask only). We evaluate 26 VLMs under a strict zero-shot setting ( T{=}0.0 for APIs, 100-token output cap); accuracy is top-1 string match. Table 1: Comprehensive Dataset Taxonomy & Texture Bias Performance. Summary of the 6 high-fidelity segmentation datasets utilizing our shape-only canonicalization. We report mean \pm standard deviation of performance for 16 leading open-weight models in High-Quality (HQ) vs. Silhouette (Sil) modes. Consistent degradation confirms universal texture dependence. Dataset Targets Geometric Focus / Visual Challenge HQ (Texture) Sil (Shape) Avg Drop Mean \pm Std (%) Mean \pm Std (%) (%) ImageNet-S 8 100 Base geometries with distinct outlines. 41.4\pm 24.7 6.2\pm 4.7 35.3 DIS5K 25 125 Complex boundaries, fine appendages. 39.8\pm 32.5 26.3\pm 20.8 13.5 ThinObject5K 17 193 Continuous, narrow geometries. 47.5\pm 29.0 46.5\pm 30.2 0.9 PASCAL VOC 7 20 Articulated objects, non-canonical views. 64.2\pm 34.4 44.9\pm 24.8 19.3 CUB-200 29 200 Fine-grained species (anatomy geometry). 10.5\pm 9.8 0.7\pm 0.8 9.8 WTP-Bench (Ours) 1,160 Extreme fine-grained associative geometry. 4.8\pm 4.6 1.2\pm 0.9 3.6 Overall Average 34.7 21.0 13.7 3 Analysis Removing RGB information induces a severe, consistent performance collapse across all architectures and datasets. Table 2: Evaluated VLMs & The Texture Bias Cliff on WTP-Bench. Master Ball (exact-match) accuracy under HQ (full-color) and Silhouette (shape-only) conditions. Relaxed tiers (Ultra Ball, Great Ball) in Table 7 . Open-weight rows are representative; full results in the Supplementary. WTP-Bench Accuracy Model Size HQ (%) Sil (%) Proprietary Models Claude Haiku 4.5 3 – 27.59 27.41 GPT-4o mini 22 – 49.31 8.53 Gemini 2.5 Flash 11 – 77.24 30.00 Claude Sonnet 4.5 3 – 65.09 53.10 Grok 4 Fast 30 – 46.90 7.93 Gemini 2.5 Pro 11 – 78.19 23.02 GPT-4.1 22 – 78.97 49.48 Representative Open-Weight Models SmolVLM 20 2B 2.24 0.95 Qwen3-VL 4B 26 4B 13.36 3.10 LLaVA-OV 7B 15 7B 3.53 1.81 LLaVA 1.5 13B 18 13B 1.81 0.69 InternVL2.5 26B 23 26B 4.14 1.72 The Texture Bias Cliff: On WTP-Bench ( Table 2 ), GPT-4.1 leads proprietary models with 78.97\% HQ accuracy but plummets to 49.48\% on silhouettes . The cliff is insurmountable for all open-weight models : Qwen3-VL 4B’s best-in-class 13.36\% HQ collapses to 3.10\% , and most architectures fall below 2.2\% . Even under relaxed evolutionary-family matching (Great Ball tier, Table 7 ), the cliff persists: GPT-4.1 drops from 84.66\% to 62.50\% ,

Read the complete text