ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis

Advik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit Das · 2025

Read ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis by Advik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit Das (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20274.txt.

Introduction

Sinha, Atreya, A, Ali, and Das teach vision-language models to read whole scenes, relations included. Their 2025 paper presents ScenarioCLIP for natural scene analysis. CLIP-style models match images with short captions well. Real scenes hold many objects acting and relating. Class lists alone miss that compositional structure.

ScenarioCLIP feeds text, grounded relations, and images together. Separate encoders handle global, object, and relation levels. Focused regions spotlight where relations happen. Contrastive cross-alignment ties the levels across modalities. An EMA teacher distills global knowledge downward steadily. The companion Action-Genome dataset supplies training material. It extends several public indoor and outdoor collections. Language models generated action and relation annotations. Manual and automatic curation cleaned the result. Benchmarks covered retrieval and fine-grained understanding tasks. Zero-shot and fine-tuned results beat prior methods. Scene-level reasoning moves beyond object naming. Robotics and surveillance both need such reading.

Catalog record

CREATORAdvik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit Das
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis Advik Sinha, Saurabh Atreya, Aashutosh A V, Sk Aziz Ali, Abhijit Das Abstract Until recently, the general corpus of CLIP-type fundamental models has widely explored either the retrieval of short descriptions or the classification of objects in the scene as SINGLE-object image classification task. The same holds for retrieving the image embedding (image retrieval task) given a text prompt. However, real-world scene images exhibit rich compositional structure involving multiple objects and actions. The latest methods in the CLIP-based literature improve class-level discrimination by mining harder negative image-text pairs and by refining permanent text prompts, often using LLMs. However, these improvements remain confined to predefined class lists and do not explicitly model relational or compositional structure. PyramidCLIP partially addresses this gap by aligning global and local visual features, yet it still lacks explicit modeling of inter-object relations. Hence, to further leverage this aspect for scene analysis, the proposed model ScenarioCLIP accepts input texts, grounded relations, and input images, along with focused regions highlighting relations to separate disentangled encoders. Feature alignment is performed with the contrastive cross-alignment, and an Exponential Moving Average (EMA)-based knowledge distillation is performed between the global level, and finer object and relation levels within each modality. The proposed ScenarioCLIP model is pretrained on curated scenario data, and finetuned for specialized downstream tasks, such as cross-modal retrieval and fine-grained visual understanding tasks. Along with the ScenarioCLIP model, to address the lack of domain-specific datasets, we generate a novel dataset by extending image-text pairs from existing diverse indoor and outdoor scenario datasets that are publicly available, such as OpenPSG, Panoptic Video Scene Graph, Kinetics-400, Multi-Moments in Time, and SemanticKITTI. We used a pipeline of existing language models to ground action, object, and relations, filled by manual and automatic curation. We established a comprehensive benchmark for several scenario-based tasks and compared it with many baseline methods. ScenarioCLIP demonstrates robust zero-shot and finetune performance on various domain-specific tasks. Our code and dataset are available at https://github.com/scenario-clip/ScenarioCLIP . 1 Introduction To date, the dual stream vision-language pre-training (DSVLPs), which is the general corpus of CLIP-type methods, has widely explored the retrieval of short descriptions or types of objects in the scene. Their popularity stems from strong few-shot and zero-shot performance, in contrast to purely visual models trained with supervised Image-Net-based classification (e.g., AlexNet Krizhevsky et al. 2012 , ResNet He et al. 2015 ), which are typically tied to a fixed label set and must be retrained or finetuned to recognise new categories. To overcome this, the paradigm of DSVLPs decouples the image encoder from the text encoder and extracts features from images and texts simultaneously, leading to simple and flexible adaptations for diverse downstream applications. However, most DSVLPs are restricted to SINGLE-object image classification or short description tasks Gao et al. 2022a ; Radford et al. 2021 ; Li et al. 2022 ; Yao et al. 2021 . A similar limitation holds while using DSVLPs for image retrieval tasks given a text prompt. However, real-world scenes may contain many objects and actions whose combinations induce complex inter-object relations. Prior DSVLP methods mainly improve class separation by: (i) mining harder image-text negatives Radenovic et al. 2023 and (ii) refining text prompts Zhou et al. 2022a , often using LLMs to synthesise or score prompt templates. While these strategies sharpen discrimination over a predefined label set, they operate at the class level and do not explicitly model relations between entities. As a result, embeddings of semantically similar pairs (e.g., person holding bag vs. person holding umbrella ) will collapse in representation space. Among recent advances, PyramidCLIP Gao et al. 2022b explores hierarchical alignment by introducing contrastive alignment between global and local features. However. PyramidCLIP does not explicitly model inter-object relations or align features as per relation semantics. In contrast, the proposed ScenarioCLIP is designed to ingest grounded relational information in scene structure using both the visual and text modalities. On the image side, the model processes the full scene image, the cropped objects and focused regions corresponding to inter-object relations . On the text side, the inputs include a global action caption for the scene, object names, and grounded relation descriptions. Each of these streams is encoded by disentangled encoders, allowing us to maintain correspondence between global context (scene/actions) and local evidence (objects and relations). The encoders are initialised with pretrained CLIP Radford et al. 2021 weights, providing a strong general-purpose prior that we then specialise to object and relation-level semantics by training on extracted object and focused regions. As illustrated in Figure 1 , real-world scenes typically contain multiple objects and relations that cannot be captured by a single global description and image-level feature. In Figure 1(a) , the scene involves a single clearly grounded relation ( motorcycle parked on street ), which can still be aligned to an appropriate global description by standard CLIP-style models, even though the relation itself remains implicit and unlabeled. In contrast, Figure 1(b) shows a scene with overlapping relations ( man sitting on motorcycle , bag on motorcycle , trees behind man ), where one global caption cannot specify which objects participate in which interactions or localise these interactions. ScenarioCLIP explicitly addresses this setting by detecting actions and objects and localising the corresponding relations between them, associating each grounded relation text with its focused region (Figure 2 ). To model the aforementioned complex scene-based relationship along with the modality-based cross-alignment, ScenarioCLIP incorporates an EMA-based knowledge-distillation within the vision-language space. We maintain an EMA-based teacher of the model and enforce consistency between global scene-level embeddings and their corresponding object and relation-level embeddings in both the visual and textual modalities. This transfer helps to stabilise high-level semantics into local representations without relying on external knowledge graphs or additional VLM teachers, in contrast to Structure-CLIP Huang et al. 2023 . Further, to address the lack of large-scale datasets with grounded actions, objects, and relations across diverse scenarios, we also introduce the Action-Genome Dataset . This proposed dataset contains approximately 600K images annotated with a global action caption, a list of object names, and multiple relation triplets (object_{1},relation,object_{2}) per image, each linked to object bounding boxes and relation-focused regions (Section 3 ). We generate these annotations via a three-stage pipeline that combines a VLM (OpenVLM/Ovis-Gemma et al. 2024 ; Lu et al. 2024 ) for action, object, and relation text, GroundingDINO Liu et al. 2023c for object grounding, and SAM Kirillov et al. 2023 plus RBF-based masking to construct relation-focused regions. In addition, we synthesise hard negative relation triplets via object swaps and antonym-based predicate substitutions, enabling contrastive learning over fine-grained relations. The resulting dataset provides rich supervision for scenario-level scene understanding and fills an important gap for training and evaluating models such as ScenarioCLIP. In summary, the contributions of this paper are threefold: • We introduce ScenarioCLIP , a relation-aware vision-language model that jointly encodes global scene context, objects, and inter-object relations within a scene, enabling multi-relation classification and object detection in complex scenes. • We construct the Action-Genome Dataset , a 600K-image corpus with action, object, and relation annotations, object bounding boxes, and relation-focused regions, providing a new dataset for scene understanding. • We propose an

Read the complete text