Prompt‑Guided Relational Reasoning for Social Behavior Understanding with Vision Foundation Models

Thinesh Thiyakesan Ponbagavathi, Chengzheng Yang, Alina Roitberg · 2025

Read Prompt‑Guided Relational Reasoning for Social Behavior Understanding with Vision Foundation Models by Thinesh Thiyakesan Ponbagavathi, Chengzheng Yang, Alina Roitberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07996.txt.

Introduction

Reading a social scene from video means knowing who is together and what the group is doing. Thinesh Thiyakesan Ponbagavathi, Chengzheng Yang, and Alina Roitberg tackle that task in a 2025 paper with a framework called ProGraD. It builds on frozen vision foundation models, whose features come from object-centered training and, dropped in naively, actually hurt group detection.

ProGraD adds a small two-layer transformer that reasons explicitly about actors and groups. Learnable group prompts nudge the frozen backbone toward social cues, while the relational decoder infers group locations, membership, and activities in one pass. Only ten million parameters are trained, under half of earlier methods. On the Cafe benchmark the system beat the prior best by clear margins, and its attention maps show which actors it grouped and why.

Catalog record

CREATORThinesh Thiyakesan Ponbagavathi, Chengzheng Yang, Alina Roitberg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-07996

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompt‑Guided Relational Reasoning for Social Behavior Understanding with Vision Foundation Models Thinesh Thiyakesan Ponbagavathi, Chengzheng Yang, Alina Roitberg Abstract Group Activity Detection (GAD) involves recognizing social groups and their collective behaviors in videos. Vision Foundation Models (VFMs), like DinoV2, offer excellent features, but are pretrained primarily on object-centric data and remain underexplored for modeling group dynamics. While they are a promising alternative to highly task-specific GAD architectures that require full fine-tuning, our initial investigation reveals that simply swapping CNN backbones used in these methods with VFMs brings little gain, underscoring the need for structured, group-aware reasoning on top. We introduce Prompt-driven Group Activity Detection (ProGraD) – a method that bridges this gap through 1) learnable group prompts to guide the VFM attention toward social configurations, and 2) a lightweight two-layer GroupContext Transformer that infers actor-group associations and collective behavior. We evaluate our approach on two recent GAD benchmarks: Cafè, which features multiple concurrent social groups, and Social-CAD, which focuses on single-group interactions. While we surpass state-of-the-art in both settings, our method is especially effective in complex multi-group scenarios, where we yield a gain of 6.5 % 6.5\% (Group mAP @ @ 1.0) and 8.2 % 8.2\% (Group mAP @ @ 0.5) using only ∼ \sim 10M trainable parameters. Furthermore, our experiments reveal that ProGraD produces interpretable attention maps, offering insights into actor‑group reasoning. Code and models will be released. Figure 1: ProGraD for Group Activity Detection. A lightweight framework that adapts vision foundation models using learnable group prompts and a minimal transformer. ProGraD infers group memberships and activities in a single pass, achieving strong results over existing methods. Introduction Group Activity Detection (GAD) ( Kim et al. 2024 ; Tamura, Vishwakarma, and Vennelakanti 2022 ; Ehsanpour et al. 2020 ; Ehsanpour et al. 2022 ) extends traditional group activity recognition by requiring models to localize multiple groups within a scene and classify their collective activities. This involves predicting who is acting, with whom, and where these groups are spatially located. This is challenging, as the model must infer group membership on the fly, while also reasoning about both individual and group-level behaviors – all in messy, real-world environments. Most existing methods ( Kim et al. 2024 ; Tamura, Vishwakarma, and Vennelakanti 2022 ) rely on fully fine-tuned CNN backbones and complex spatiotemporal group reasoning modules, making them costly to train and reliant on heuristic group formation strategies. Vision foundation models ( Bommasani et al. 2021 ) (VFMs) like Dino v2 ( Oquab et al. 2023 ) and CLIP ( Radford et al. 2021 ) provide generalizable visual features, and are increasingly adapted to downstream tasks ( Barın, Aydemir, and Güney 2024 ; Kumar et al. 2024 ; An et al. 2024 ; Zhang et al. 2022 ) using lightweight probing ( Chen et al. 2020 ) , where a small set of parameters is trained on top of frozen VFM backbones. This raises a natural question: Can we recognize social dynamics GAD with VFMs directly? . Unfortunately, our experiments show that naively swapping CNN backbones with VFMs in existing GAD methods yields negligible performance gains. This aligns with prior analyses indicating that VFMs, primarily trained on object-centric data, struggle to capture fine-grained multi-object relations and social interactions critical for GAD ( Wagner and Harmeling 2025 ) . Likewise, existing probing methods ( van den Oord, Li, and Vinyals 2019 ; Lee et al. 2019 ; Ponbagavathi and Roitberg 2025 ) are primarily designed for classification or retrieval and lack the structured reasoning required for actor-group associations and collective behavior modelling. Motivated by this, we propose ProGraD - Pro mpt-driven Gr oup A ctivity D etection – a lightweight model placed on top of frozen VFMs that adapts them specifically for social behaviour understanding by 1) injecting learnable group prompts and 2) introducing a minimal two‑layer GroupContext Transformer (GCT). Unlike simple probing heads, GCT serves as a robust decoder, effective in both backbone frozen and fully fine‑tuned modes, scaling from efficiency‑focused adaptation to high‑capacity settings. The prompts steer the backbone’s attention toward socially meaningful regions, mitigating the object‑centric bias of pretrained VFMs. GCT then models actor–group associations and collective activities through its Grouping and Contextual Attention layers. This modular design enables simultaneous group localization, membership inference, and joint action recognition in a single pass- with only \sim 10M trainable parameters, less than half of prior state-of-the-art methods. Moreover, our experiments reveal that its structured attention layers provide interpretable attention maps, making actor–group reasoning explicit and visually traceable. We evaluate ProGraD on two recent GAD benchmarks: Café ( Kim et al. 2024 ) , featuring multiple concurrent social groups, and Social-CAD ( Ehsanpour et al. 2020 ) , where most groups are singletons. On Café, ProGraD achieves a 17.03 Group mAP @ 1.0 and 39.11 Group mAP @ 0.5, outperforming prior state-of-the-art by 6.5 and 8.2 points, respectively, with further gains when fully fine‑tuned. We also obtain the highest Outlier mIoU of 67.89, indicating precise outlier (singleton) localization. On Social‑CAD, ProGraD achieves a state‑of‑the‑art 70.00% social accuracy and 88.38% membership accuracy, surpassing all prior methods, and further improves to 70.11% social accuracy and 89.29% membership accuracy when fully fine‑tuned, establishing new benchmarks on both metrics. These gains, particularly in complex multi‑group scenes, underscore ProGraD’s strength in socially‑aware group reasoning. To summarize, our key contributions are : (1) We introduce ProGraD, the first framework to effectively adapt pretrained VFMs for Group Activity Detection by incorporating learnable group prompts, bridging the gap between object‑centric pretraining and multi‑agent social reasoning. (2) We design a minimal two-layer GroupContext Transformer that enables efficient, interpretable reasoning over social formations, supporting group localization, membership inference, and joint activity recognition. (3) We demonstrate that ProGraD achieves state-of-the-art results on Café and Social‑CAD using only \sim 10M trainable parameters, less than half of prior methods. Related Work Group Activity Detection Group Activity Detection (GAD) extends traditional group activity recognition ( Ehsanpour et al. 2022 ; Tamura, Vishwakarma, and Vennelakanti 2022 ; Zhou et al. 2022 ) by requiring models to localize multiple groups within a scene and classify each group’s activity. This involves both group membership prediction and group-level activity classification. Early approaches ( Ehsanpour et al. 2022 ; Han et al. 2022b ; Li et al. 2022 ) employed graph neural networks (GNNs) with spectral clustering to model actor relationships.While effective in structured environments, these methods relied on non-task-specific clustering algorithms, leading to slow inference and limited adaptability in dynamic real-world scenes. Recent transformer-based methods address these shortcomings by unifying localization and reasoning. HGC ( Tamura, Vishwakarma, and Vennelakanti 2022 ) performs group localization in the 2D coordinate space by applying Deformable DETR ( Carion et al. 2020 ) to match actors to predicted group centers using spatial cues. In contrast, Practical GAD introduces learnable group tokens that enable matching in the embedding space, allowing the model to reason over semantic similarity rather than relying solely on spatial proximity. Other transformer-based group activity recognition (GAR) approaches ( Gavrilyuk et al. 2020 ; Han et al. 2022a ; Li et al. 2021 ) also rely on CNN backbones to extract frame-level features, which are then processed through stacked spatial-temporal transformer blocks to model inter-actor relations and group-level dynamics. However, the limited

Read the complete text