Spot the Ball : A Benchmark for Visual Social Inference

Neha Balamurugan, Sarah Wu, Adam Chun, Cristobal Eyzaguirre, Tobias Gerstenberg · 2025

Read Spot the Ball : A Benchmark for Visual Social Inference by Neha Balamurugan, Sarah Wu, Adam Chun, Cristobal Eyzaguirre, Tobias Gerstenberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00261.txt.

Introduction

Neha Balamurugan, Sarah Wu, Adam Chun, Gabe Gaw, Cristobal Eyzaguirre, and Tobias Gerstenberg test whether vision-language models can read social cues. People infer hidden facts from gaze, pose, and body direction with ease. The benchmark, Spot The Ball, removes the ball from sports photos in soccer, basketball, and volleyball, then asks where it was.

Humans beat four leading models by a wide margin, scoring two to three times higher. Models lean on shallow habits, such as guessing near the center or near players. Humans instead follow eyes and body orientation. The gap suggests future models need built-in ways to encode behavior cues, not just raw pattern matching.

Catalog record

CREATORNeha Balamurugan, Sarah Wu, Adam Chun, Cristobal Eyzaguirre, Tobias Gerstenberg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00261

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Spot the Ball : A Benchmark for Visual Social Inference Neha Balamurugan, Sarah Wu, Adam Chun, Cristobal Eyzaguirre, Tobias Gerstenberg Abstract Humans excel at visual social inference , the ability to infer hidden elements of a scene from subtle behavioral cues such as other people’s gaze, pose, and orientation. This capacity drives everyday social reasoning in humans and is critical for developing more human-like AI agents. We introduce Spot the Ball , a challenging benchmark for evaluating visual social inference in vision–language models (VLMs) using sports as a test domain. The task is to localize a removed sports ball from soccer, basketball, and volleyball images. We present a curated evaluation set with human baselines and a scalable pipeline for generating additional test items. We evaluate four state-of-the-art VLMs (Gemini, GPT, LLaMA, Qwen) using three prompting strategies, finding that humans are consistently two to three times more accurate ( 20 20 – 34 % 34\% ) than models ( ≤ 17 % \leq 17\% ) across all sports. Our analyses show that models rely on superficial spatial heuristics—such as guessing near the image center or nearby players—while humans leverage social cues like gaze direction and body pose. These findings reveal a persistent human–model gap in visual social reasoning and underscore the need for architectures that explicitly encode structured behavioral cues to achieve robust, human-like inference. Figure 1 : Overview of the Spot the Ball task. Given an image with the ball removed, humans and models infer the likely location by reasoning about player pose and gaze. Models are prompted under three conditions, whereas humans receive only the base prompt. 1 Introduction When someone scans the floor with narrowed eyes, we infer they must be searching for something. When a friend approaches us with open arms, we anticipate a hug. As humans, we readily use subtle behavioral cues, such as gaze, pose, and orientation to infer implicit information. This ability is rooted in our theory of mind (ToM), the capacity to reason about others’ beliefs, desires, and intentions to predict their behavior ( 5 ; 10 ; 41 ) . ToM is fundamental to everyday human interaction ( 3 ; 12 ; 37 ) , and increasingly critical for AI systems deployed in social contexts: a robot nurse that misinterprets a patient’s gesture, or an autonomous car that fails to anticipate a pedestrian’s intention, may behave unsafely. Figure 2 : Pipeline for constructing the Spot the Ball dataset . We retrieve and filter sports footage from YouTube by alignment to the prompts, detect players and balls using an object detector, and inpaint the ball region with stable diffusion before overlaying a 6×10 grid for location annotation. While LLMs perform some types of social reasoning based on text relatively well ( 13 ) , such behavior may rely primarily on linguistic pattern matching rather than perceptual grounding. Further, human social reasoning rarely occurs through text alone—we continuously integrate visual cues such as gaze direction, facial expressions, body language, and spatial relationships alongside (or in the absence of) verbal information. To evaluate whether AI systems can perform similar visually-grounded social inference, we study VLMs. VLMs process the visual information available to human observers and are increasingly deployed in embodied, safety-critical contexts where robust interpretation of visual social cues is essential. Existing benchmarks for social reasoning either evaluate models on fully visible scenes ( 28 ; 46 ) or emphasize inanimate objects ( 24 ) , failing to capture how humans reason about others under partial information. We introduce Spot the Ball , inspired by a classic newspaper puzzle, to evaluate whether VLMs can infer hidden objects from visual social cues in real-world images. Ball sports provide an ideal testbed as players’ gaze, posture, and positioning are causally coupled to the ball, yielding interpretable social signals. We use static images to isolate social reasoning from motion dynamics. We contribute: (1) 150 curated sports images with human baselines, (2) systematic evaluation of four leading VLMs under multiple prompting strategies, and (3) a scalable pipeline for generating ball sports reasoning tasks, with which we have produced 3,000 additional soccer images for training and analysis. We provide the first structured evaluation of VLMs’ ability to leverage social cues for inferring hidden objects in real-world scenes. Our results reveal large, consistent gaps between human and model performance. 2 Related Work Human social reasoning. Humans acquire social reasoning abilities early in life, learning from infancy to represent and infer others’ beliefs, desires, and intentions ( 41 ; 36 ; 40 ) . Adults readily infer rich social states from minimal visual cues—moving geometric shapes ( 14 ; 39 ) , gaze ( 18 ) , facial expressions ( 38 ) , and body language ( 9 ) . As AI systems are increasingly deployed in social contexts, they must exhibit similar reasoning capabilities for seamless human interaction ( 27 ) . Social reasoning benchmarks for AI. Most social reasoning benchmarks for AI operate in text-only settings, evaluating ToM ( 44 ; 25 ; 13 ) , empathy ( 8 ; 15 ) , moral reasoning ( 26 ; 34 ) , deception detection ( 7 ; 20 ) , and negotiation ( 1 ; 33 ) . While informative, these benchmarks cannot assess whether models extract social cues from visual perception—the primary modality through which humans perform social inference. Video-based benchmarks extend social reasoning evaluation to dynamic visual contexts, including theory-of-mind in human interactions ( 16 ; 35 ; 23 ) and synthetic simulations ( 11 ) , causal and counterfactual video QA ( 45 ; 43 ) , and social interaction understanding ( 47 ; 42 ) . Static image benchmarks are less common but include Visual Commonsense Reasoning ( 48 ) , facial expression understanding ( 29 ; 19 ) , and social relationship recognition ( 22 ) . However, existing visual benchmarks either present fully observable scenes ( 28 ; 46 ) or focus on physical occlusion without social cues ( 24 ; 30 ) . No prior work evaluates whether models can infer hidden information purely from visual social cues. Our contribution. We introduce a zero-shot, image-based benchmark requiring models to infer a hidden object by integrating pose, gaze, and orientation cues from human agents. Unlike prior work, our task isolates visually-grounded social reasoning in naturalistic scenes with partial information, reflecting how humans actually perform social inference in everyday contexts. Figure 3 : Player density and coverage across sports in the Spot the Ball dataset. In Soccer frames (A) player count and area are intermediate compared to the other sports. Volleyball frames (B) feature the most players but each occupies a smaller visual area, providing weaker pose and gaze cues. Basketball (C) has fewer yet larger players, offering clearer postural and gaze information. 3 Spot the Ball In Spot the Ball , the objective is to infer the location of a removed ball in a sports frame (see Figure 1 ). This task evaluates a model’s ability to localize a hidden object through reasoning over social and physical contextual cues such as players’ gaze, body orientation, and spatial positioning, rather than relying on direct visual evidence of the object itself in addition to sport specific knowledge.

Read the complete text