SigmaCollab : An Application-Driven Dataset for Physically Situated Collaboration
Dan Bohus, Sean Andrist, Ann Paradiso, Nick Saw, Tim Schoonbeek, Maia Stiber · 2025
Read SigmaCollab : An Application-Driven Dataset for Physically Situated Collaboration by Dan Bohus, Sean Andrist, Ann Paradiso, Nick Saw, Tim Schoonbeek, Maia Stiber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02560.txt.
Introduction
Dan Bohus, Sean Andrist, Ann Paradiso, Nick Saw, Tim Schoonbeek, and Maia Stiber release a dataset of people being coached through physical tasks by an AI assistant in mixed reality. Eighty-five sessions record untrained participants following a head-mounted agent's guidance through procedural work in the real world. The recordings are rich: the participant's and the system's audio, the view from the headset, depth maps, and tracking of head, hands, and gaze, plus annotations added afterwards.
The collection is small, about fourteen hours, and the authors are upfront about that. Its value lies in being application-driven and interactive, which surfaces research problems that scripted datasets hide: how an assistant should speak, watch, and intervene while someone works with their hands. They plan to build benchmarks for mixed-reality task assistance on top of it.
Catalog record
| CREATOR | Dan Bohus, Sean Andrist, Ann Paradiso, Nick Saw, Tim Schoonbeek, Maia Stiber |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02560
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SigmaCollab : An Application-Driven Dataset for Physically Situated Collaboration Dan Bohus, Sean Andrist, Ann Paradiso, Nick Saw, Tim Schoonbeek, Maia Stiber Abstract We introduce SigmaCollab , a dataset enabling research on physically situated human-AI collaboration. The dataset consists of a set of 85 sessions in which untrained participants were guided by a mixed-reality assistive AI agent in performing procedural tasks in the physical world. SigmaCollab includes a set of rich, multimodal data streams, such as the participant and system audio, egocentric camera views from the head-mounted device, depth maps, head, hand and gaze tracking information, as well as additional annotations performed post-hoc. While the dataset is relatively small in size ( ≈ \approx 14 hours), its application-driven and interactive nature brings to the fore novel research challenges for human-AI collaboration, and provides more realistic testing grounds for various AI models operating in this space. In future work, we plan to use the dataset to construct a set of benchmarks for physically situated collaboration in mixed-reality task assistive scenarios. SigmaCollab is available at https://github.com/microsoft/SigmaCollab . 1 Introduction Fluid human-machine interaction has been a decades-long pursuit in the research community. Building computational systems that can engage in interaction and collaborate with people in the physical world , such as virtual assistive agents, interactive robots, or mixed-reality assistants brings forward multiple challenges at the intersection of several research fields: artificial intelligence, computer vision, natural language processing, and human-computer interaction. Physically situated systems need to make sense (in real time and to the extent needed by each specific application) of the surrounding physical environment and of the actions performed by humans in the environment. This often involves a set of competencies such as self-localization and mapping, object detection and tracking, depth estimation, face detection, body pose tracking, action recognition, and so forth. Many of these tasks have been extensively studied in the computer vision research community, with significant progress over the last two decades. The recent emergence of foundation models and large multimodal models has enabled a transition towards open-domain computer vision, and, by tying the models to language, has enabled reasoning about and answering more complex questions about the physical world. A lot of this progress has been made possible by the collection and curation of various datasets and data resources that have been used in the community to train and to comparatively evaluate the performance of various algorithms. Situated collaboration raises, however, additional interaction-related challenges. In addition to understanding the environment, objects, and actions, creating seamless situated collaborations requires fine-grained temporal coordination that is grounded in a deep understanding of human cognitive states such as, intentions, goals, beliefs, and various affective states. In these areas, progress has been slower. Today, most systems still use a "ping-pong" model of interaction where one user question is followed in a linear manner by one system answer, which limits naturalness and curtails opportunities for fluid, mixed-initiative collaboration. Part of the difficulty with interaction-related challenges lies in reifying, formalizing, and measuring the cognitive states and interactive phenomena involved. Moreover, the traditional approach of using collected, static datasets to build and evaluate models works only to a certain degree when it comes to interaction. Any trained model, once deployed, may alter user behavior, shifting the distribution of the underlying data. The study and evaluation of interactive phenomena are best done in the context of actual interactive systems. To foster and enable more research on interaction and collaboration challenges, we introduce a new resource: an application-driven, interactive dataset called SigmaCollab . The dataset consists of 85 interactive sessions in which untrained participants interacted with a mixed-reality assistive AI application which guided them in performing certain tasks in the physical world, such as binding a notebook or installing the wheels on a skateboard. The dataset is egocentric and multimodal in nature; it includes audio, egocentric camera views, head, hands and gaze tracking, and it spans approximately 14 hours of interaction data. SigmaCollab ’s value-proposition lies in its application-driven and interactive nature. The data collection approach, in which participants interact with an AI application surfaces novel challenges that are not present in non-interactive datasets. As users engage with the system while pursuing meaningful and varied goals, the approach yields data with greater ecological validity. Finally, the open-source nature of the AI application used for collecting the data enables researchers to deploy models developed or evaluated based on this data in the context of the target application, and allows them to collect additional data to study their end-effects on metrics of overall task performance. We publish SigmaCollab to catalyze rigorous, application-grounded study of fluid human–AI collaboration and to close the gap between lab benchmarks and real-world performance. 2 Related Work Publicly released academic datasets have historically played a transformative role in advancing computer vision research. The release of ImageNet [ 17 ] was a turning point for large-scale visual classification, enabling the development of powerful deep convolutional architectures. Subsequently, COCO [ 27 ] significantly pushed the field of object detection and segmentation, while Kinetics [ 12 ] has become a cornerstone for video understanding and action recognition, and Cityscapes [ 13 ] for pixel-level scene understanding. Beyond these general-purpose datasets, application-driven benchmarks have similarly accelerated progress in specialized domains. For example, MNIST [ 25 ] catalyzed early breakthroughs in digit recognition and thereby modern deep learning, KITTI [ 19 ] has been instrumental in autonomous driving research, and LFW [ 23 ] in face recognition. These examples highlight that introducing a well-designed, meaningful dataset in a specific application area can catalyze rapid advances in methodology, benchmarks, and downstream impact that can shape the direction of entire research fields. Spurred by an increased interest in embodied interaction, mixed-reality and robotics, multiple datasets and benchmarks have been developed in recent years for egocentric computer vision settings. For example, the OpenEQA dataset [ 28 ] provides embodied question-answer pairs that are anchored in videos (or 3D-scans) of real world environments. The EPIC-KITCHENS project [ 14 , 15 , 16 ] captured daily activities in the kitchen from a first-person perspective, in support of challenges like action detection, action recognition, object segmentation, etc. The Ego4D dataset [ 20 ] includes over 3000 hours spanning hundreds of different indoors and outdoors activities, and defines various benchmarks for memory, forecasting and social understanding challenges. Later, the Ego-Exo4D [ 21 ] effort extends this by simultaneously capturing both ego and exo-centric views of skilled human activities and includes benchmarks for egocentric tracking of body and hand poses and for proficiency estimation. Other datasets like GTEA Gaze, GTEA Gaze+, EGTEA Gaze+ [ 18 , 26 ] focused on capturing eye gaze in conjunction with egocentric views, and leveraging that signal on various action-related challenges. A number of other egocentric datasets have also been constructed around structured, procedural task execution or performance (rather than spontaneous activities), such as Assembly101 [ 33 ] , CaptainCook4D [ 30 ] , IndustReal [ 32 ] , MECCANO [ 4 ] , etc. These datasets support challenges like mistake detection, procedure step recognition, active object detection, action anticipation. While these egocentric datasets provide a rich empirical testbed for many computer vision challenges, they generally capture a single actor performing an activity in the world, and