Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation
Maëlic Neau Zoe Falomir Cédric Buche Akihiro Sugimoto · 2025
Read Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation by Maëlic Neau Zoe Falomir Cédric Buche Akihiro Sugimoto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01209.txt.
Introduction
Scene graph generation turns an image into a graph of objects and the relations between them, such as a person riding a bike. Maelic Neau, Zoe Falomir, Cedric Buche, and Akihiro Sugimoto tackle the open-vocabulary form, where models must handle a wide range of relations, not a fixed short list. Current benchmarks use narrow vocabularies, so they judge such models poorly.
The team proposes a reference-free metric, a score that needs no gold labels, to judge relation prediction fairly. They also build high-quality synthetic training data through region-specific prompt tuning of vision-language models. Tests show pre-training on that new data improved how well models generalized. The work gives the field both a sharper ruler and better material to learn from.
Catalog record
| CREATOR | Maëlic Neau Zoe Falomir Cédric Buche Akihiro Sugimoto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation Maëlic Neau Zoe Falomir Cédric Buche Akihiro Sugimoto Abstract Scene Graph Generation (SGG) encodes visual relationships between objects in images as graph structures. Thanks to the advances of Vision-Language Models (VLMs), the task of Open-Vocabulary SGG has been recently proposed where models are evaluated on their functionality to learn a wide and diverse range of relations. Current benchmarks in SGG, however, possess a very limited vocabulary, making the evaluation of open-source models inefficient. In this paper, we propose a new reference-free metric to fairly evaluate the open-vocabulary capabilities of VLMs for relation prediction. Another limitation of Open-Vocabulary SGG is the reliance on weakly supervised data of poor quality for pre-training. We also propose a new solution for quickly generating high-quality synthetic data through region-specific prompt tuning of VLMs. Experimental results show that pre-training with this new data split can benefit the generalization capabilities of Open-Voc SGG models 1 1 1 Code and data available at https://github.com/Maelic/OpenVocSGG . 1 Introduction The task of Scene Graph Generation (SGG) has gained growing interest in recent years. SGG aims at generating a structured abstract representation of image content as a directed acyclic graph composed of <subject,predicate,object> triplets, where each subject and object is traditionally grounded to an image region through 2D coordinates (bounding boxes). Applications of such representations include Visual-Question Answering (VQA) [ 22 ] , Image Captioning [ 29 ] , or even Robotics [ 19 ] . While a considerable amount of work has tackled the task of SGG in a closed-vocabulary fashion [ 32 , 24 , 27 ] , approaches targeting the task in open-vocabulary settings are scarce [ 5 , 8 ] . Open-Vocabulary SGG (OV-SGG) is a critical task to enable a new range of applications in robotics [ 1 ] or embodied agent perception [ 16 ] , where high-quality annotated data can be difficult to acquire. Similar to other Open-Vocabulary tasks [ 7 ] , OV-SGG approaches are evaluated on closed-vocabulary benchmarks (Visual Genome [ 13 ] , PSG [ 27 ] ) in a zero-shot manner. However, current benchmarks only evaluate models on a small number of relations (i.e., <50 ), which limits our understanding of the generalization capabilities of such models. This paper aims to extend the metrics used in OV-SGG with the introduction of a reference-free evaluation to grade the efficiency of models where there is a lack of groundtruth for data annotations. OV-SGG approaches often rely on a large amount of data for pre-training. Recent work proposes to generate pre-training data for OV-SGG through caption-image regions alignment [ 30 ] or directly through the use of state-of-the-art Vision-Language Models (VLMs) [ 26 ] . This paper shows that the former approaches results in low-quality coarse annotations, which may hinder the pre-training of OV-SGG models. Regarding the latter approach, the current literature is missing a comprehensive review of the abilities of VLMs to generate high-quality, consistent, and diverse visual relations. This paper proposes an extensive investigation of the performance of different VLMs for the prediction of fine-grained visual relationships. We apply the results of our investigation to generate high-quality synthetic relationship annotations for 200K images. We call this new dataset the Fine-Grained Open-Vocabulary SGG dataset (FG-OV SGG). In a series of experiments, we show that our generated data is of better quality than the existing previous data in the literature. This paper describes the following contributions: 1. A new reference-free metric, the RelCLIPScore, to evaluate OV-SGG models and VLMs in unconstrained settings (Section 3 ); 2. A quantitave evaluation on the VLM performance for region-specific relation prediction, highlighting their strengths and weaknesses for the task (Section 4 ); and 3. A qualitative evaluation providing a new dataset, FG-OV SGG, generated using VLMs evaluated using RelCLIPScore metrics, which can be beneficial to Open-Vocabulary SGG models (Section 5 ). 2 Related Work This section presents the most recent works regarding open-vocabulary SGG metrics and weakly annotated data. Open-Vocabulary SGG Metrics. Traditional approaches evaluate Open-Vocabulary Relationship Detection and Open-Vocabulary Scene Graph Generation using mean Average Precision (mAP) [ 30 ] or Recall@K/meanRecall@K [ 8 , 11 ] metrics on standard SGG benchmarks [ 13 , 27 ] in a zero-shot manner. The Recall@K evaluates the performance of a model on the top predictions (i.e. top 20/50/100) for each image, independent of the class. The meanRecall@K evaluates the performance on the average of all predicate classes, which is more significant for long-tail learning such as in the task of SGG. However, using ground truth references to evaluate Open-Vocabulary models is not optimal for the following reasons: 1. Relations-based datasets are not extensively annotated, leaving a considerable amount of true relations unannotated (i.e. miss positive samples [ 34 ] ) 2. The SGG task includes a multi-label classification paradigm since multiple predicates can be correct for a given <subject,object> pair [ 24 ] . 3. Evaluating the task of SGG with ranking metrics does not give a good overview of the model performance. For instance, a true positive predicted at rank 1 and 99 will lead to the exact same Recall@100 score. In a recent work [ 4 ] , the SGScore has been proposed to evaluate the alignment of Image Generation models with complex scenes. The SGScore computes the similarity between an image and a graph by asking multiple-choice questions to an ”oracle” VLM (such as: What is the relationship between the person and the sports ball in the image? A) kicking; B) throwing; C) holding; D) no visible relationship. [ 4 ] ). The authors of SGScore do not give any details on the choice of negative examples for the question-answer pairs which hindered reproducibility. Also, multiple-choice questions can not account for multi-label instances, which are overly present in Open-Vocabulary settings. In order to solve these challenges, we propose to tackle the task of Open-Vocabulary SGG in a reference-free manner, similar to what has been done in the task of Image Captioning [ 10 ] . By taking advantage of Vision-Language Models such as CLIP [ 20 ] , we propose to evaluate the alignment between relation triplets and corresponding image regions formed by the union of subject and object. Weakly Annotated Data. In contrast to other tasks, such as Object Detection [ 7 , 23 ] , the task of SGG has not yet seen the emergence of large-scale, high-quality datasets with a large vocabulary. As a result, the main approaches in OV-SGG are using image caption supervision for pre-training [ 21 , 9 ] and then evaluating zero-shot transfer on closed-source datasets such as Visual Genome [ 13 ] or PSG [ 27 ] . However, captions-only supervision can limit the understanding of SGG models of fine-grained relations. Several recent approaches have been proposed to alleviate this bias: (i) the RLIPv2 [ 30 ] was proposed to generate weakly annotated data by grounding captions to image regions using a dedicated R-TAGGER model; and (ii) the All-Seeing Project V2 (ASv2) [ 26 ] proposed to use Vision-Language Models (i.e. GPT4) to generate open-vocabulary objects and relations annotations on new images. Specifically RLIPv2 [ 30 ] proposed to generate pseudo-annotations for