ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios
Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Daniele Di Mauro, Camillo Quattrocchi, Alessandro Passanisi, Irene D’Ambra, Antonino Furnari, Giovanni Maria Farinella · 2026
Read ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios by Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Daniele Di Mauro, Camillo Quattrocchi, Alessandro Passanisi, Irene D’Ambra, Antonino Furnari, Giovanni Maria Farinella (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09741.txt.
Introduction
Francesco Ragusa and a large team release ENIGMA-360, a data set of worker activity filmed in a real industrial setting. Paired head worn and fixed camera videos, time matched in both views, carry time and space labels. The set supports three tasks: splitting actions in time, naming key steps and spotting hand object contact. Current top methods struggle on all three tasks.
Catalog record
| CREATOR | Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Daniele Di Mauro, Camillo Quattrocchi, Alessandro Passanisi, Irene D’Ambra, Antonino Furnari, Giovanni Maria Farinella |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55435 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09741
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Daniele Di Mauro, Camillo Quattrocchi, Alessandro Passanisi, Irene D’Ambra, Antonino Furnari, Giovanni Maria Farinella Abstract Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial environments and enhance their safety. However, progress in this area is hindered by the lack of datasets capturing both views in realistic industrial scenarios. To address this gap, we propose ENIGMA-360, a new ego-exo dataset acquired in a real industrial scenario. The dataset is composed of 180 egocentric and 180 exocentric procedural videos temporally synchronized offering complementary information of the same scene. The 360 videos have been labeled with temporal and spatial annotations, enabling the study of different aspects of human behavior in industrial domain. We provide baseline experiments for 3 foundational tasks for human behavior understanding: 1) Temporal Action Segmentation, 2) Keystep Recognition and 3) Egocentric Human-Object Interaction Detection, showing the limits of state-of-the-art approaches on this challenging scenario. These results highlight the need for new models capable of robust ego-exo understanding in real‑world environments. We publicly release the dataset and its annotations at https://fpv-iplab.github.io/ENIGMA-360/ . keywords Ego-exo dataset, Industrial domain, Egocentric and Exocentric views, Human Behavior Understanding 1 Introduction Figure 1 : Overview of the ENIGMA-360 dataset. On the left, we show the spatial annotations available. On the top-right, we report the designed procedure for acquisition purpose. On the bottom-right we show synchronized egocentric and exocentric video frames illustrating the multi-view setup. Understanding the human behavior and its interactions with the physical world is a fundamental aspect for building AI agents capable to support humans in daily life. Procedures often involve complex, multi-step activities that require diverse skills and the use of different objects. For instance, in industrial environments, tasks such as machinery maintenance demand a sequence of coordinated actions. Following standard protocols, a worker may begin by wearing personal protective equipment (i.e., PPEs) such as helmets or gloves, proceed to interact with tools (e.g., take the iron solder, connect the battery to the electric screwdriver), perform functional tests (e.g., press buttons on the electric panel), and conclude by reporting the intervention (e.g., take the pen, write the report). These scenarios highlight the need for intelligent systems that can temporally segment complex procedures, identify the occurrence and ordering of individual keysteps, and detect detailed human–object interactions, enabling real-time verification of task execution quality and safety compliance. To effectively assist humans in complex real-world environments, an ideal intelligent system should be able to understand human behavior from multimodal observations. These observations can be captured by wearable smart cameras (e.g., smart glasses) 55 ; 13 ; 49 , which provide detailed insights into human-object interactions and performed actions, as well as by fixed cameras that capture the broader environmental context 27 . These complementary viewpoints (ego-exo) offer a holistic understanding of the same scene, enabling intelligent assistants to provide comprehensive, 360-degree support (see Figure 1 -bottom-right). For example, during the execution of a maintenance procedure, the intelligent system should continuously monitor the sequence of worker actions, understand the order in which steps are performed, and track the progression of the procedure in real time. Additionally, the system should verify whether each step has been completed correctly, provide guidance on how to perform it properly, and alert the worker to potential safety risks (e.g., Before touching the soldering iron, wear protective gloves! ). However, despite this potential, existing datasets rarely provide synchronized ego-exo recordings in real industrial environments. In general, the design of novel architectures able to understand human behavior has been driven by the availability of public datasets spanning diverse domains 27 ; 64 ; 4 ; 45 . These include everyday environments such as kitchens 14 ; 46 ; 79 , daily activities 54 ; 35 , and simulated industrial settings 63 ; 58 ; 61 ; 6 . However, existing industrial-like datasets typically rely on proxy scenarios such as toy assemblies or simplified environments with textureless components 58 ; 63 ; 61 , due to the inherent challenges of capturing data in real industrial contexts, such as privacy regulations, safety constraints, and the need to protect proprietary processes. As a result, they fail to capture the visual complexity, tool variability, and procedure realism required for developing deployable industrial-assistance models. To address these limitations, we present ENIGMA-360, a new multi-view dataset composed of 180 egocentric and 180 exocentric videos acquired in a real industrial laboratory. The video pairs are temporally synchronized, with each pair capturing the same scene from two distinct perspectives. This dual-view setup enables the extraction of complementary information: the egocentric view offers fine-grained details of hand-object interactions, while the exocentric view provides broader contextual understanding of the environment and task dynamics. To collect the dataset, we set up an industrial laboratory equipped with real industrial tools and components, such as a power supply unit, soldering iron, and electrical boards. In collaboration with industrial experts, we designed two realistic maintenance procedures to ensure the dataset reflects realistic industrial workflows. To eliminate the need for physical manuals 58 ; 63 ; 61 , we developed a wearable application that provides step-by-step instructions to participants during data collection. ENIGMA-360 is richly annotated to support a wide range of research tasks from both egocentric and exocentric perspectives. Specifically, it includes temporal annotations marking the start and end times of keysteps, as well as spatial annotations identifying hand-object interactions within keyframes. In addition to the manual annotations, we provide a comprehensive set of additional resources to support extended research tasks. Specifically, we release segmentation masks for hands and manipulated objects 30 , and provide DINOV2 video features 52 . Furthermore, to encourage research on synthetic-to-real transfer and scalable model training, we make available the 3D models of the laboratory and all industrial objects present in the dataset. This combination of synchronized multi-view videos, detailed annotations, and 3D assets is unique among industrial datasets. To highlight the usefulness of the proposed multi-view dataset, we performed baseline experiments related to 3 tasks focused on human behavior understanding: 1) Temporal Action Segmentation, 2) Keystep Recognition and 3) Egocentric Human-Object Interaction Detection. The performance gap observed across methods highlights the substantial domain gap between industrial and everyday activities and confirms that ENIGMA-360 poses new challenges for human behavior understanding. This work extends our previous research 59 by incorporating 129 additional egocentric videos and 180 exocentric videos, enriching the dataset with new temporal and spatial annotations, and introducing a benchmark tailored to the analysis of human behavior in industrial environments. We temporally synchronize each ego-exo pair to enable joint modeling across viewpoints, and we provide fine-grained temporal labels of keysteps along with spatial annotations capturing human–object interactions. These include bounding boxes around the user’s hands and manipulated objects, as well as explicit hand–object association links. In addition, we expand the benchmark by evaluating state‑of‑the‑art methods under this more challenging setting. This extended study contributes a new suite of experiments spanning three additional tasks and offers further insights into the complexity and unique characteristics of industrial scenarios. In sum the contributions of this work are as follows: 1) we present ENIGMA-360,