EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms
Brian VanVoorst, Nicholas Walczak, Christopher Gilleo, Charles Meissner, Fabio Felix, Iran Roman, Bea Steers, Claudio Silva, Yuhan Shen, Zijia Lu, Shih-Po Lee, Ehsan Elhamifar · 2026
Read EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms by Brian VanVoorst, Nicholas Walczak, Christopher Gilleo, Charles Meissner, Fabio Felix, Iran Roman, Bea Steers, Claudio Silva, Yuhan Shen, Zijia Lu, Shih-Po Lee, Ehsan Elhamifar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22036.txt.
Introduction
VanVoorst and eleven colleagues introduce EgoMAGIC, an egocentric medical activity dataset from DARPA's Perceptually-enabled Task Guidance program. The dataset comprises 3,355 videos of 50 medical tasks, with at least 50 labeled videos per task. The medical training data has been released with an action detection challenge focused on eight medical tasks. From this dataset, 40 YOLO models were trained using about 2 million labels to detect 124 medical objects. The paper presents baseline action detection results for the eight tasks across three models, with the best method achieving average mAP 0.526 across the eight tasks.
Catalog record
| CREATOR | Brian VanVoorst, Nicholas Walczak, Christopher Gilleo, Charles Meissner, Fabio Felix, Iran Roman, Bea Steers, Claudio Silva, Yuhan Shen, Zijia Lu, Shih-Po Lee, Ehsan Elhamifar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35445 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22036
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EgoMAGIC- An Egocentric Video Field Medicine Dataset for Training Perception Algorithms Brian VanVoorst, Nicholas Walczak, Christopher Gilleo, Charles Meissner, Fabio Felix, Iran Roman, Bea Steers, Claudio Silva, Yuhan Shen, Zijia Lu, Shih-Po Lee, Ehsan Elhamifar Abstract This paper introduces EgoMAGIC (Medical Assistance, Guidance, Instruction, and Correction), an egocentric medical activity dataset collected as part of DARPA’s Perceptually-enabled Task Guidance (PTG) program. This dataset comprises 3355 videos of 50 medical tasks , with at least 50 labeled videos per task . The primary objective of the PTG program was to develop virtual assistants integrated into augmented reality headsets to assist users in performing complex tasks. To encourage exploration and research using this dataset, the medical training data has been released along with an action detection challenge focused on eight medical tasks. The majority of the videos were recorded using a head-mounted stereo camera with integrated audio. From this dataset, 40 YOLO models were trained using 1.95 million labels to detect 124 medical objects , providing a robust starting point for developers working on medical AI applications. In addition to introducing the dataset, this paper presents baseline results on action detection for the eight selected medical tasks across three models, with the best-performing method achieving average mAP 0.526. Although this paper primarily addresses action detection as the benchmark, the EgoMAGIC dataset is equally suitable for action recognition, object identification and detection, error detection, and other challenging computer vision tasks. The dataset is accessible via zenodo.org (DOI: 10.5281/zenodo.19239154). 1 Introduction Figure 1 : Visual summary of the dataset showing the total number of videos and tasks, along with unique annotated labels for steps, actions, and objects. The images illustrate the recording environments (professional or in-lab), highlight the five most challenging tasks with keywords descriptions of their steps, and provide examples of objects related to specific actions. The advances in machine learning video-related tasks are heavily related to the development and enhancement of powerful deep learning architectures, such as convolutional neural networks [ 23 ] and transformers [ 5 ] , along with large-scale datasets, such as Youtube-8M [ 17 ] , Something-something [ 15 ] , Kinetics [ 18 ] , EPIC Kitchens [ 7 ] , and Ego4D [ 12 ] . Although these and other datasets help models achieve strong generalization, more specialized datasets are still needed in areas like medical procedures. This paper introduces the EgoMAGIC dataset (Medical Assistance, Guidance, Instruction, and Correction), an egocentric dataset designed to train deep learning models specifically for medicine tasks. The dataset and benchmark models were developed as part of the Defense Advanced Research Projects Agency’s (DARPA) Perceptually-enabled Task Guidance (PTG) program [ 2 ] that aimed to create real-time augmented reality (AR) assistants that guide users through tasks. These assistants are intended to recognize medical procedures, automatically track completed steps, and provide guidance on the next step or corrections when steps are missed or performed incorrectly. This scenario presents exciting opportunities for the deep learning community. To encourage research and exploration of the EgoMAGIC dataset, the action detection component (identifying the start and stop time of individual steps) has been emphasized as a stand-alone challenge, which is the focus of this paper. This challenge is structured similarly to other egocentric action detection datasets, such as EPIC Kitchens [ 7 ] and the Trauma THOMPSON challenge [ 19 ] . Novelty & Contributions. Emergency response medical triage is inherently fast-paced and chaotic, occurring in unstructured environments with significant open-world variability [ 25 ] . PTG selected combat medicine as a domain due to its unique challenges and the significant value an AR assistant could offer to caregivers in high-stress emergencies. In this scenario, the EgoMAGIC dataset introduces several unique action detection challenges not found in other egocentric datasets, such as: 1. Some steps within many tasks are extremely brief, lasting as little as a second or less (see Table 1 and Figure 2 ). 2. Steps can frequently be performed concurrently, leading to overlapping activities. 3. Some steps are optional, allowing for variability in step sequence and completion. 4. Rapid egomotion is common throughout the activities, adding complexity to the visual data. 5. Realistic clutter of objects and environment. Figure 1 top- and bottom-center present examples of a partial view of objects. While the primary focus is on action detection—identifying the start and stop times of steps in medical procedures—the dataset also supports other research challenges, including action recognition (identifying the action being performed within a given clip), and action anticipation (predicting future actions). Furthermore, the dataset contains extensive object labels for medical equipment and supplies, making it valuable for object detection research. The dataset includes 3355 videos covering 50 medical tasks as highlighted in Figure 1 . For data, it provides: • 8.2M frames extracted from the left lens of the Zed stereo camera and 1.1 extracted from the HL2 main camera • Over 1.95 million labeled objects across 124 object classes • More than 17,000 task step annotations across 286 step classes • Over 39,000 hand-object interactions The dataset also provides 40 pre-trained YOLOv8 [ 8 ] models to help developers with object detection and action recognition tasks. In addition to introducing a new dataset, this paper presents the approaches and results of research teams that developed deep learning models for action detection in the PTG scope. The results work as a benchmark and challenge report over the EgoMAGIC dataset. Task (8) Num Steps Num Videos Avg Dur (sec) Avg Step Dur (s) Min Step Dur (s) % of Steps with Overlaps NPA Tube [A8] 5 109 45.12 5.12 0.81 0.0% Apply Tourniquet [M2] 8 138 51.31 4.01 0.37 16.3% Pressure Dressing [M3] 5 50 82.88 15.98 0.43 15.1% Wound Packing [M4] 3 54 98.97 32.15 2.41 29.6% X-Stat [M5] 5 45 41.37 7.55 0.65 0.0% Ventilate (BVM) [R16] 5 49 41.54 6.85 0.43 1.1% Apply Chest Seal [R18] 5 69 44.02 8.55 0.13 14.8% Needle Chest Decomp [R19] 6 65 81.15 9.72 0.43 4.7% Table 1 : Summary of dataset tasks used for the tests. Figure 2 : Average step duration in seconds for the 8 selected tasks. 2 Related Datasets Egocentric Action Detection. Identifying and detecting actions in egocentric videos is important for AR assistants to understand the environment and guide users through their tasks, even potentially catching user mistakes. However, working with the egocentric view is challenging, as it includes camera motion, occlusions, and overlapping activities that can hinder the reliability of action detection. There is a growing effort to provide datasets and benchmarks that help developers and researchers enhance deep learning models for AR assistants. For example, Georgia Tech Egocentric Activity Dataset (GTEA) [ 21 , 24 ] and YouHome Activities-of-Daily-Living (ADL) [ 9 ] , focused on daily activities and offered a glimpse into how people interact with objects in their immediate surroundings. EPIC-Kitchens [ 7 ] and others built upon it, expanded the domain of egocentric vision to cooking activities in home kitchens, allowing researchers to examine fine-grained hand-object manipulations within a complex and cluttered environment. Something-Something-v2 (SSv2) [ 15 ] shows users performing basic daily actions, the objects involved in each action, and labels with simple descriptions defined as caption templates. Ego4D [ 12 ]