Open World Scene Graph Generation using Vision Language Models
Amartya Dutta, Kazi Sajeed Mehrab, Medha Sawhney, Abhilash Neog, Mridul Khurana, Sepideh Fatemi, Aanish Pradhan, M. Maruf, Ismini Lourentzou, Arka Daw, Anuj Karpatne · 2025
Read Open World Scene Graph Generation using Vision Language Models by Amartya Dutta, Kazi Sajeed Mehrab, Medha Sawhney, Abhilash Neog, Mridul Khurana, Sepideh Fatemi, Aanish Pradhan, M. Maruf, Ismini Lourentzou, Arka Daw, Anuj Karpatne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08189.txt.
Introduction
Dutta and co-authors generate scene graphs for images from any domain without task training, in a 2025 computer vision paper. A scene graph lists the objects in a picture and the relations between pairs. Classic systems learn those relations from labeled datasets, which ties them to fixed object lists. Their framework instead queries pretrained vision-language models directly, with structured prompts and embedding checks.
A light refinement step cleans up the proposed object pairs. The team also formalize an open-world test protocol where neither objects nor relations were seen in training data. On three benchmarks, the training-free system produced sensible relational graphs for unseen categories. The authors pitch pretrained model knowledge as a substitute for expensive scene graph annotation.
Catalog record
| CREATOR | Amartya Dutta, Kazi Sajeed Mehrab, Medha Sawhney, Abhilash Neog, Mridul Khurana, Sepideh Fatemi, Aanish Pradhan, M. Maruf, Ismini Lourentzou, Arka Daw, Anuj Karpatne |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08189
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open World Scene Graph Generation using Vision Language Models Amartya Dutta, Kazi Sajeed Mehrab, Medha Sawhney, Abhilash Neog, Mridul Khurana, Sepideh Fatemi, Aanish Pradhan, M. Maruf, Ismini Lourentzou, Arka Daw, Anuj Karpatne Abstract Scene-Graph Generation (SGG) seeks to recognize objects in an image and distill their salient pairwise relationships. Most methods depend on dataset-specific supervision to learn the variety of interactions, restricting their usefulness in open-world settings, involving novel objects and/or relations. Even methods that leverage large Vision Language Models (VLMs) typically require benchmark-specific fine-tuning. We introduce Open-World SGG, a training-free, efficient, model-agnostic framework that taps directly into the pretrained knowledge of VLMs to produce scene graphs with zero additional learning. Casting SGG as a zero-shot structured-reasoning problem, our method combines multimodal prompting, embedding alignment, and a lightweight pair-refinement strategy, enabling inference over unseen object vocabularies and relation sets. To assess this setting, we formalize an Open-World evaluation protocol that measures performance when no SGG-specific data have been observed either in terms of objects and relations. Experiments on Visual Genome, Open Images V6, and the Panoptic Scene Graph (PSG) dataset demonstrate the capacity of pretrained VLMs to perform relational understanding without task-level training. † † footnotetext: ‡ Denotes equal contribution, † denotes equal advising and * denotes the corresponding authors : {amartya, karpatne}@vt.edu 1 Introduction Scene Graph Generation (SGG) aims to convert an image into a structured graph, where nodes correspond to object entities and edges capture the semantic relationships between them. This intermediate representation enables structured reasoning over visual content and has been shown to benefit a range of downstream tasks, including image captioning, visual question answering, and referring expression generation Johnson et al. 2015 ; Teney et al. 2017 ; Hudson and Manning 2019 ; Yang et al. 2019 . Achieving accurate SGG requires a strong understanding of both visual appearance and the contextual semantics that govern interactions between objects. Traditional approaches to SGG are predominantly supervised, trained on datasets like Visual Genome that provide dense annotations of object and relationship triplets. While these models have made significant progress, they are fundamentally constrained by the scope and vocabulary of the annotated data. Annotation is labor-intensive and often biased, and the long-tail distribution of object and predicate classes further hampers generalization to complex, real-world imagery Zellers et al. 2018 ; Lu et al. 2016 . To address these limitations, recent efforts have explored Open-Vocabulary SGG (OV-SGG), where the goal is to predict objects or relations that were not seen during training. This includes tasks such as Open-Vocabulary Object Detection (OVD) – detecting unseen object categories - and Open-Vocabulary Relationships (OVR) – predicting unseen predicates between known object pairs Gu et al. 2021 . More recent formulations consider fully open-world settings where objects or predicates may be novel at inference time Chen et al. 2024 ; Liu et al. 2025 . However, even these models rely on fine-tuning or auxiliary training stages, limiting their adaptability and requiring curated data. Given the rapid advancements of Vision-Language Models (VLMs) trained on massive image-text corpora Bai et al. 2023 ; Deitke et al. 2024 ; Liu et al. 2024b ; OpenAI 2024 ; Team et al. 2024 , a natural question arises: Can VLMs enable zero-shot scene graph generation without requiring task specific training? These models demonstrate strong visual and language generalization, and recent work has proposed reformulating SGG sub-tasks – particularly predicate classification – as image-text matching problems. Yet, most of these approaches still incorporate dataset-specific modules or rely on expensive pairwise inference pipelines, therefore not evaluating VLMs on a truly open-world setting. Despite the growing interest in leveraging VLMs for SGG, progress in evaluating these models for SGG has been hindered by several key limitations. First , there is a lack of standardized baselines for open-world SGG, making it difficult to assess how well models generalize to unseen categories. Second , there is no established methodology for prompting VLMs to generate scene graphs in a way that is both effective and comparable with existing methods. Third , the open-ended nature of VLM outputs makes it nontrivial to elicit structured predictions like subject–predicate–object triplets, which would enable their outputs to be quantitatively evaluated with popular SGG datasets, like Visual Genome Krishna et al. 2017 , Panoptic Scene Graph (PSG) Yang et al. 2022 and OpenImage (OI) Kuznetsova et al. 2020 . To bridge this gap, we present Open World SGG (OwSGG) an end-to-end, model-agnostic framework for zero-shot scene graph generation using pretrained VLMs. Our method combines multimodal prompting, embedding alignment, and a lightweight pair-refinement strategy to transform raw VLM outputs into structured scene graphs that are compatible with existing evaluation protocols, enabling quantitative benchmarking without requiring any task-specific training. Using this framework, we conduct a comprehensive evaluation of two popular VLMs, LLaVa-next Liu et al. 2024b and Qwen2-VL Wang et al. 2024 – across a range of settings – including closed vocabulary, open-vocabulary objects and open-vocabulary relationships. We further introduce a fully open-world case. While we do not obtain the best results in the closed-world setting, we find that VLMs, despite no access to task-specific training, can match or even surpass these models in certain open-world cases. To encourage further research, we introduce an open-world baseline that isolates the performance of VLMs on novel object AND novel relation pairs, providing a new point of comparison for future methods. Our findings highlight the potential of pretrained vision-language models for scalable scene graph understanding and underscore the need for new methods and benchmarks tailored to the open-world setting. 2 Related Works Scene Graph Generation introduced in Krishna et al. 2017 , aims to localize, classify and predict relationships between entities in images, enabling structured visual understanding. Early works in SGG ( Johnson et al. 2015 ; Xu et al. 2017 ) predict pairwise relationships between objects to construct graphs and demonstrate various scene graph applications, such as Image Captioning and VQA ( Hildebrandt et al. 2020 ; Hudson and Manning 2019 ) . ( Zellers et al. 2018 ) improve SGG by higher-order “motifs” in the object–predicate–object statistics, yielding large gains. However, these methods rely on fully supervised training with labeled scene graph data, which is expensive to annotate, difficult to scale to the wide variety of objects and relationships that can exist in natural scenes, and can suffer from data imbalance. To mitigate these, weakly and semi-supervised approaches Kim et al. 2024b and imbalanced learning strategies Cong et al. 2023 have been explored, but they remain restricted to a closed vocabulary of relationships seen during training. Recent work has introduced open-vocabulary SGG (OV-SGG), in two primary settings: Open-vocabulary Detection (OvD), which predicts known predicates between unseen object categories Chen et al. 2024 ; He et al. 2022 ; Zareian et al. 2021 ; Zhang et al. 2023a , and Open-vocabulary Relationships (OvR), which classifies unseen predicates between known object categories Chen et al. 2024 ; Zareian et al. 2021 . A more recent line of work addresses the combined OvD+OvR setting, though still relying on task-specific supervision Chen et al. 2024 ; Zhang et al. 2023a . Tackling a truly open-world, training-free SGG setting – where models generate scene graphs without any fine-tuning or supervision was previously infeasible due to limitations in model