BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding
Patrick Knab Orgest Xhelili Inis Buzi Drago Andres Guggiana Nilo, [-0.2ex] Mohd Saquib Khan Lorenz Kolb Manuel Scherzer Kerem Yildirir, [-0.2ex] Christian Bartelt Philipp Johannes Schubert, [-0.2ex] Ramblr.ai Research Technical University of Clausthal Equal contribution. · 2026
Read BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding by Patrick Knab Orgest Xhelili Inis Buzi Drago Andres Guggiana Nilo, [-0.2ex] Mohd Saquib Khan Lorenz Kolb Manuel Scherzer Kerem Yildirir, [-0.2ex] Christian Bartelt Philipp Johannes Schubert, [-0.2ex] Ramblr.ai Research Technical University of Clausthal Equal contribution. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12074.txt.
Introduction
Patrick Knab and nine colleagues build BARISTA, a rich test set for first-person visual understanding. It holds 185 real videos of coffee making, by machine, by hand-held filter, and by capsule, each frame tagged with what is seen, where it is, and what hands do. From these tags they build hands-free tests: find a phrase in the scene, name hand actions, point to items, name steps, link items, and answer time-based questions. Scores swing widely across test types, and no one model family leads them all.
Catalog record
| CREATOR | Patrick Knab Orgest Xhelili Inis Buzi Drago Andres Guggiana Nilo, [-0.2ex] Mohd Saquib Khan Lorenz Kolb Manuel Scherzer Kerem Yildirir, [-0.2ex] Christian Bartelt Philipp Johannes Schubert, [-0.2ex] Ramblr.ai Research Technical University of Clausthal Equal contribution. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72983 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12074
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding Patrick Knab Orgest Xhelili Inis Buzi Drago Andres Guggiana Nilo, [-0.2ex] Mohd Saquib Khan Lorenz Kolb Manuel Scherzer Kerem Yildirir, [-0.2ex] Christian Bartelt Philipp Johannes Schubert, [-0.2ex] Ramblr.ai Research Technical University of Clausthal Equal contribution. Abstract Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks. We introduce BARISTA , a densely annotated egocentric dataset and benchmark of 185 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps. From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding. Code and dataset available at https://huggingface.co/datasets/ramblr/BARISTA . Figure 1: BARISTA overview. BARISTA is an egocentric coffee-preparation dataset and benchmark spanning three preparation styles, dense task-relevant annotations, and object-focused and semantic evaluation tasks derived from a shared scene graph. 1 Introduction Vision-language models (VLMs) increasingly serve as the perception backbones of agentic and embodied systems operating in the physical world [ 11 , 23 , 15 , 3 , 41 , 2 ] . Such systems require a coordinated set of capabilities: localization of task-relevant objects and regions, interpreting interactions with the environment, recognizing ongoing activities in context, and reasoning about how the scene evolves over time [ 41 , 20 , 24 , 8 ] . These abilities span multiple levels of abstraction, from masks and boxes to typed relations and process-level reasoning, and are difficult to assess when each capability is benchmarked on a different dataset under its own assumptions and evaluation protocol. This points to a core benchmark-design requirement: evaluations must support controlled, fine-grained analysis of failure across levels, not merely report aggregate scores. Most benchmarks isolate a single aspect of compositional understanding, such as activity recognition, long-video question answering, or object-centric perception [ 32 , 27 , 13 , 37 ] . As a result, failures are hard to interpret: errors on process-level tasks may stem from weak localization, interaction understanding, relational reasoning, or temporal modeling, without a way to disentangle them. This is particularly problematic given recent evidence that VLMs remain brittle even on basic spatial cognition [ 2 , 22 , 14 , 35 ] . Existing egocentric and instructional video datasets offer a path to tackling this—providing dense spatial annotations, activity labels, or relational structure—but each typically covers only part of the required supervision. Because their domains, label spaces, and evaluation setups differ, they provide limited common ground to trace a high-level reasoning failure back to a concrete perceptual gap. We introduce Barista , a densely annotated egocentric dataset and a unified benchmark of 185 real-world coffee-preparation videos designed to address this gap. The technical core of BARISTA is a per-frame co-registered scene graph. Each annotated frame includes instance masks, bounding boxes, per-instance attributes, and directed typed relations anchored to persistent object identities, together with activity and process-step labels. We deliberately trade duration for annotation density and traceability: the 4.4 hours of BARISTA carry 3.61\mathrm{M} instance masks, 4.33\mathrm{M} attributes, and 2.48\mathrm{M} typed relations, all co-registered through persistent identities. This structure supports high-quality multi-task annotation at practical scale and naturally enables query-driven VLM evaluation. More importantly, it makes BARISTA explicitly decomposable: object-, interaction-, relation-, and process-level tasks are defined over the same underlying evidence, enabling controlled comparisons and hypothesis testing about which lower-level failures drive higher-level errors. Coffee preparation serves here as a compact yet expressive procedural testbed: it is structured enough to require process reasoning, varied enough across machine types to expose differences in tools, interfaces, interaction patterns, and step order, and familiar enough that failure modes are easy to inspect. Figure 1 provides an overview of BARISTA, illustrating the covered coffee-preparation styles, dense annotation layers, and the benchmark tasks. For object-based perception, we provide annotations for phrase grounding, hand-object interaction recognition, segmentation masks, and referring expression generation; for semantic scene understanding, we focus on activity recognition, relation extraction, and temporal visual question answering. The unified benchmark evaluates language-facing tasks under a shared zero-shot protocol, while the released annotations additionally support downstream training and fine-tuning of specialist perception models. Because all language-facing benchmark items are derived deterministically from this verified graph, failures on process-level tasks can be traced back on the same frames to the localization, interaction, and relation annotations probed by the other tasks. We benchmark four VLM families — Qwen [ 46 ] , Gemma [ 42 ] , GPT [ 1 ] , and Gemini [ 41 ] — across different model sizes, finding that no model performs uniformly best across the different tasks. Notably, spatial grounding performance varies far more across models than semantic recognition, and the capabilities underlying localization-heavy tasks are largely orthogonal to those driving temporal QA. Our main contributions are as follows: • We introduce Barista , a densely annotated egocentric dataset of 185 coffee-preparation videos built around a per-frame co-registered scene graph linking masks, boxes, attributes, relations, hand-object interactions, and procedural steps via persistent object identities. • We propose a semi-automated annotation pipeline and a graph-based task-generation procedure for dense spatial, relational, interaction, and language-derived supervision. • We define a unified, decomposable benchmark with standardized zero-shot prompting and task-specific metrics, and use it to show that current frontier and open-weight VLMs exhibit task-orthogonal capability profiles. 2 Related work Egocentric and instructional video datasets. Large-scale egocentric resources such as EPIC-KITCHENS [ 9 ] , Ego4D [ 16 ] , Ego-Exo4D [ 17 ] , and the long-form EgoLife [ 48 ] have substantially advanced first-person video understanding, primarily through narration-derived action labels and temporal boundaries rather than per-frame structured annotations. VISOR [ 10 ] extends EPIC-KITCHENS with 272\mathrm{K} manual pixel masks for hands and active objects across 257 classes, 9.9\mathrm{M} interpolated dense masks, and 67\mathrm{K} hand-object contact-state relations, but does not annotate typed semantic relations between objects, per-instance attribute vocabularies, or activity segments. Table 1: Comparison of selected related video datasets. sparse : objects annotated only at movement events. contact : hand-object contact only. fixture : object-fixture associations from 3D lifting. typed : semantic object-object or hand-object relation labels beyond boxes/masks. Dataset View Dataset Size Instance Masks Bounding Boxes Object Attributes Typed Relations Action Labels Temporal Tracks EPIC-KITCHENS [ 9 ] Ego 100h ✓ VISOR (EPIC) [ 10 ] Ego 36h ✓ ✓ contact ✓ HD-EPIC [ 34 ] Ego 41h sparse ✓ fixture ✓ ✓ Ego4D [ 16 ] Ego 3000h ✓ ✓ Ego-Exo4D [ 17 ] Ego+Exo 1286h ✓ ✓ Assembly101 [ 38 ] Ego+Exo 513h ✓ EPFL-SmartKit. [ 5 ] Ego+Exo 29.7h ✓ PVSG [ 49 ] Ego+Exo 9h ✓ ✓ ✓ ✓ BARISTA (Ours) Ego 4.4h ✓ ✓ ✓ ✓ ✓