Inference-Time Structural Reasoning for Compositional Vision-Language Understanding

Amartya Bhattacharya · 2026

Read Inference-Time Structural Reasoning for Compositional Vision-Language Understanding by Amartya Bhattacharya (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27349.txt.

Introduction

Bhattacharya tests whether scene graphs help vision-language models reason about composition at inference time. Models often confuse captions that share words but differ in relations. Four diverse models are benchmarked on Winoground with and without scene graph input. A dependency parser extracts subject-relation-object triples from each caption. A graph asymmetry scorer then injects the structural priors by optimal matching. Qwen3-VL-8B-Thinking reached a group score of 62.75, far above encoder models. Multi-turn scene graph filtering lifted it further to 66.0.

Catalog record

CREATORAmartya Bhattacharya
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22959
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inference-Time Structural Reasoning for Compositional Vision-Language Understanding Amartya Bhattacharya Abstract Vision-language models (VLMs) excel at image–text retrieval yet persistently fail at compositional reasoning , distinguishing captions that share the same words but differ in relational structure. We present a unified evaluation and augmentation framework benchmarking four architecturally diverse VLMs , CLIP, BLIP, LLaVA, and Qwen3-VL-8B-Thinking , on the Winoground benchmark under plain and scene-graph-augmented regimes. We introduce a dependency-based TextSceneGraphParser (spaCy) extracting subject–relation–object triples, and a Graph Asymmetry Scorer using optimal bipartite matching to inject structural relational priors. Caption ablation experiments (subject/object masking and swapping) reveal that Qwen3-VL-8B-Thinking achieves a group score of 62.8 , far above all encoder-based models, while a proposed multi-turn SG filtering strategy further lifts it to 66.0 , surpassing prior open-source state-of-the-art. We analyze the capability–augmentation tradeoff and find that SG augmentation benefits already-capable models while providing negligible or negative gains for weaker baselines. Code: https://github.com/amartyacodes/Inference-Time-Structural-Reasoning-for-Compositional-Vision-Language-Understanding 1. Introduction Modern VLMs achieve remarkable performance on image captioning, visual question answering, and cross-modal retrieval. Yet a fundamental question persists: do these models truly understand the compositional structure of language grounded in visual scenes? Consider the minimal pair: “The dog is chasing the cat” vs. “The cat is chasing the dog.” Both share identical content words; they differ solely in the agent–patient assignment. A model relying on bag-of-words statistics assigns nearly identical scores to both captions for any image, failing this basic compositional test. The Winoground benchmark ( Thrush et al. 2022 ) operationalizes this challenge through 400 carefully curated minimally contrastive image–caption pairs (Figure 1 ). Initial evaluations revealed CLIP performing only marginally above random chance, motivating the search for structural augmentation strategies. Our work is motivated by three observations. First, architectural diversity matters : dual-encoder, cross-attention, and generative VLMs process compositional information through different mechanisms. Second, explicit structure can help : scene graphs provide a structural representation absent from raw embeddings. Third, integration must be architecture-appropriate : an additive score prior suits embedding models; prompt injection suits generative models. 1.1. Contributions. (1) A unified evaluation framework for four VLMs on Winoground; (2) a TextSceneGraphParser covering SVO, prepositional, copular, existential, and possessive constructions; (3) a Graph Asymmetry Scorer via optimal bipartite matching targeting agent–patient role reversal; (4) systematic caption ablation experiments quantifying entity-identity vs. relational-syntax reliance; (5) a multi-turn SG filtering strategy achieving group score 66.0, a new open-source state-of-the-art on Winoground. Figure 1 : Winoground examples. Captions share the same words arranged differently; images share the same visual elements interacting differently. Models must correctly pair each caption with its image. Figure 2 : TextSceneGraphParser architecture (spaCy rule-based). Five parallel extraction rules produce triples that are deduplicated and cached per caption string. Figure 3 : Caption ablation pipeline. spaCy identifies semantic spans offline; captions are transformed (mask/swap subject, object, or both) and evaluated without SG injection. 2. Related Work 2.1. Compositional reasoning benchmarks. Thrush et al. 2022 introduced Winoground, demonstrating that all VLMs tested performed near random chance on compositional image–caption matching. Diwan et al. 2022 analyzed failure modes, categorizing them as visual difficulty, linguistic ambiguity, and genuine compositional failure. Concurrently, Yuksekgonul et al. 2023 introduced the ARO benchmark (Attribution, Relation, Order) with over 50,000 test cases from Visual Genome and COCO, demonstrating that contrastive VLMs systematically act as bag-of-words models , for example, BLIP assigns higher probability to “the grass is eating the horse” over the correct caption with 81% confidence. Hsieh et al. 2024 followed with SugarCrepe , identifying biases in ARO’s hard negative construction and proposing LLM-generated fluent negatives across seven edit categories (replace/swap/add object, attribute, relation) that drive blind text models to chance accuracy. 2.2. Inference-time compositional augmentation. A key family of methods improves compositional reasoning at inference time without retraining. Jiang et al. 2024 proposed ComCLIP , a training-free approach that disentangles input images into subject, object, and action sub-images and performs evolving matching over compositional embeddings, mitigating spurious entity co-occurrence biases in CLIP, SLIP, and BLIP-2 across Winoground, VL-Checklist, SVO, and ComVG. Mitra et al. 2024 introduced CCoT (Compositional Chain-of-Thought, CVPR 2024), a zero-shot prompting method that first instructs the model to generate a scene graph from the input image, then uses that graph as a structured intermediate reasoning step , improving compositional performance on Winoground, WHOOPS!, and general multimodal benchmarks without fine-tuning. Our multi-turn SG is closely related to both: like ComCLIP we operate at inference time, and like CCoT we leverage scene graph structure, but we use text scene graphs extracted from captions (requiring no visual SG generator) and inject them via a formally defined asymmetry scoring mechanism or multi-turn self-filtering. 2.3. Training-time compositional fine-tuning. Yuksekgonul et al. 2023 proposed NegCLIP , fine-tuning CLIP with POS-tag-based hard negative captions, achieving substantial ARO improvements with minimal downstream degradation. Mishra et al. 2025 proposed SCRAMBLe (ICCV Workshops 2025), generating compositional hard negatives via chain-of-thought LLM prompting and adversarial filtering, then applying direct preference optimization to Molmo-7B , achieving 54.8% group score on Winoground, the prior open-source state-of-the-art. Herzig et al. 2023 proposed SGVL , incorporating Visual Genome scene graph annotations into CLIP fine-tuning via learnable graph tokens that attend to patch embeddings, improving compositional performance across multiple benchmarks. Our method surpasses SCRAMBLe (66.0 vs. 54.8%) without any retraining, positioning inference-time structural reasoning in sufficiently capable base models as a compelling alternative to preference tuning. 2.4. Large multimodal models. The shift to billion-parameter generative VLMs has produced striking Winoground gains. Liu et al. 2023 showed LLaVA-1.5-13B achieves 36.5% group via yes/no scoring. Lin et al. 2024 introduced VQAScore , using a VQA model to compute image–text alignment as the probability of “yes” to “Does this image show {caption}?”, providing richer compositional signal than cosine similarity. Most recently, Wu et al. 2025 showed that reframing evaluation as a global matching problem (GroupMatch) and applying test-time self-training enables SigLIP-B16 to reach 67% group score and GPT-4.1 to surpass human performance (91.4%) , revealing that standard GroupScore metrics systematically underestimate hidden model capabilities. 2.5. Scene graphs in VL understanding. Scene graphs , directed graphs encoding objects, attributes, and relations , have been used for image retrieval ( Johnson et al. 2015 ) , VQA ( Shi et al. 2019 ) , and image generation ( Johnson et al. 2018 ) . Text scene graphs parse captions into triple structures analogously. Our TextSceneGraphParser is most closely related to spaCy-based parsers used in SGVL ( Herzig et al. 2023 ) and CCoT ( Mitra et al. 2024 ) , but operates purely on caption text with explicit handling of five syntactic construction types. 3. Winoground Benchmark Each example (I_{0},I_{1},c_{0},c_{1},\tau) consists of two images, two captions sharing the same content words but differing in relational structure, and a linguistic-phenomenon tag. Given scoring function s:\mathcal{I}\times\mathcal{C}\to\mathbb{R} , we compute four scores s_{00},s_{10},s_{01},s_{11} and define: Text \displaystyle=\mathbb{1}[s_{00}>s_{10}]\cdot\mathbb{1}[s_{11}>s_{01}] (1) Image \displaystyle=\mathbb{1}[s_{00}>s_{01}]\cdot\mathbb{1}[s_{11}>s_{10}] (2) Group \displaystyle=\text{Text}\cdot\text{Image} (3) Random-chance baselines: Text = Image = 0.25, Group \approx 0.063 . 4. Models and Scoring Functions We evaluate four VLMs spanning the major architectural paradigms (Table 1 ). Table 1 : Base scoring functions per model. Model Architecture Scoring CLIP ViT-B/32 Dual encoder CLS cosine BLIP ITM Base Cross-attention P(\text{match}) LLaVA 1.5-7B Generative decoder P(\text{yes}) Qwen3-VL-8B Generative decoder (MoE) EOS cosine CLIP ( Radford et al. 2021

Read the complete text