ACT360: An Efficient 360-Degree Action Detection and Summarization Framework for Mission-Critical Training and Debriefing

1 Given Name Surname, 2 Given Name Surname, Aditi Tiwari, Klara Nahrstedt · 2025

Read ACT360: An Efficient 360-Degree Action Detection and Summarization Framework for Mission-Critical Training and Debriefing by 1 Given Name Surname, 2 Given Name Surname, Aditi Tiwari, Klara Nahrstedt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12852.txt.

Introduction

Aditi Tiwari and Klara Nahrstedt present ACT360, a system that watches 360-degree training video and writes the debrief itself, in a 2025 paper. Disaster, military, and industrial drills are usually reviewed by hand from flat video, slowly and with blind spots. Their detector, 360YOWO, adds spatial attention and distortion-aware convolution tuned to panoramic warping.

Shrinking tricks, quantization and pruning, cut the model by 74 percent while accuracy slipped only 1.5 points, and inference sped up. On 55 labeled panoramic videos across seven drill actions, the system held up in varied conditions. A web explorer then retrieves actions and drafts text summaries with a large language model. The authors pitch the framework for any high-stakes training review.

Catalog record

CREATOR1 Given Name Surname, 2 Given Name Surname, Aditi Tiwari, Klara Nahrstedt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-12852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ACT360: An Efficient 360-Degree Action Detection and Summarization Framework for Mission-Critical Training and Debriefing 1 Given Name Surname, 2 Given Name Surname, Aditi Tiwari, Klara Nahrstedt Abstract Effective training and debriefing are critical in high-stakes, mission-critical environments such as disaster response, military simulations, and industrial safety, where precision and minimizing errors are paramount. The traditional post-training analysis relies on manually reviewing 2-dimensional (2D) videos, a time-consuming process that lacks comprehensive situational awareness. To address these limitations, we introduce ACT360, a system that leverages 360-degree videos and machine learning for automated action detection and structured debriefing. ACT360 integrates 360YOWO , an enhanced You Only Watch Once (YOWO) model with spatial attention and equirectangular-aware convolution (EAC) to mitigate panoramic video distortions. To enable deployment in resource-constrained environments, we apply quantization and model pruning, reducing the model size by 74% while maintaining robust accuracy (mAP drop of only 1.5%, from 0.865 to 0.850) and improving inference speed. We validate our approach on a publicly available dataset of 55 labeled 360-degree videos covering seven key operational actions, recorded across various real-world training sessions and environmental conditions. Additionally, ACT360 integrates 360AIE (Action Insight Explorer) , a web-based interface for automatic action detection, retrieval, and textual summarization using large language models (LLMs), significantly enhancing post-incident analysis efficiency. ACT360 serves as a generalized framework for mission-critical debriefing, incorporating EAC, spatial attention, summarization, and model optimization. These innovations apply to any training environment requiring lightweight action detection and structured post-exercise analysis. I Introduction Mission-critical training, including firefighter drills, military exercises, and emergency response simulations, relies on accurate debriefing to evaluate performance and enhance decision-making. Traditionally, training sessions are recorded using fixed 2D cameras, requiring instructors to manually review hours of footage to identify key actions and mistakes. This manual review process is time-consuming, error-prone, and lacks comprehensive situational awareness. In high-stakes environments where every second matters, there is a pressing need for automated action detection and summarization to improve training efficiency and effectiveness. 360-degree videos provide a promising alternative by capturing the entire operational environment, eliminating blind spots, and enabling immersive analysis. However, analyzing 360-degree videos presents significant challenges that existing 2D action detection methods fail to address. Fine-grained action recognition is required to detect complex interactions rather than just identifying objects or individuals. Geometric distortions from equirectangular projection (ERP) introduce stretching and warping, particularly near the poles, which degrade the accuracy of traditional convolutional networks. Additionally, the higher resolution of 360-degree videos significantly increases computational overhead, making efficient processing difficult. Despite advances in deep learning-based action detection models like You Only Watch Once (YOWO) [ 1 ] , standard approaches do not account for ERP distortions, leading to inaccurate action localization. Recent works such as InternVideo2 [ 2 ] and 2D Conv-RBM + LSTM [ 3 ] have improved temporal action localization and spatiotemporal learning but are designed for rectilinear videos and require extensive adaptation for panoramic data. Similarly, TokenLearner [ 4 ] improves computational efficiency in video classification but lacks the spatial localization capabilities necessary for detecting fine-grained actions in dynamic environments. These limitations highlight the need for a specialized framework capable of addressing the unique challenges of 360-degree action detection. Furthermore, mission-critical environments require a lightweight, resource-efficient solution that can run on standard computing hardware without relying on large-scale cloud processing. Many existing models prioritize computational efficiency for classification (TokenLearner), hand-object interaction (EffHandNet) [ 5 ] , or general video understanding (InternVideo2), but none are designed for resource-efficient, end-to-end 360-degree action detection. Therefore, adapting existing 2D models or classification-based frameworks is not a viable solution for accurate action detection in resource-constrained 360-degree mission-critical training scenarios. To address these limitations, we present ACT360, a novel debriefing and action detection system for mission-critical training. ACT360 integrates multiple components: (1) 360YOWO (Enhanced YOWO for 360-degree video), a custom action detection model incorporating Equirectangular-Aware Convolution (EAC) and 360-degree Spatial Attention to mitigate ERP distortions and improve accuracy; (2) 360AIE (Action Insight Explorer), a web-based debriefing interface that provides automatic action detection, retrieval, and textual summarization of key events in training videos using large language models (LLMs); (3) Optimized Model Deployment, where quantization and pruning significantly reduce computational complexity, enabling deployment on resource-constrained hardware such as standard laptops with consumer GPUs; (4) We provide a publicly available 360-degree firefighter training dataset with 55 annotated videos of real-world firefighter drills, including ladder operations, door breaching, and hose deployment, available on the project website. While 360YOWO is validated on firefighter training, the techniques developed in this system can be extended to various fields, including disaster response, military simulations, medical training, and law enforcement. By curating and annotating new datasets, ACT360 can be adapted to other mission-critical training environments, ensuring that action detection remains accurate, efficient, and deployable on resource-constrained devices. Key Innovations. This work introduces the following key innovations: 1. Custom 360-Degree Firefighting Dataset: We introduce a novel dataset of annotated 360-degree firefighting videos, capturing diverse actions across various environmental conditions. Developed with firefighting instructors, it provides a foundation for action detection in emergency response. Unlike existing datasets such as 360Action [ 6 ] , which focus on general actions, ours specifically targets mission-critical firefighter training. 2. Enhanced YOWO for 360-Degree Video (360YOWO): We introduce 360YOWO, a custom action detection model incorporating EAC and 360-degree Spatial Attention to handle the unique challenges of equirectangular projection (ERP). This enables the model to focus on relevant regions within the panoramic view while mitigating distortions inherent in 360-degree footage. While attention mechanisms have been used in viewport prediction, 360YOWO is specifically designed for fine-grained action detection, identifying subtle yet critical actions in dynamic, high-risk scenarios. 3. Equirectangular-Aware Convolution (EAC): We develop a custom convolution operation that dynamically adjusts based on latitude coordinates, effectively addressing the warping effects near the poles of equirectangular projections and enabling more accurate feature extraction across the entire 360-degree field of view. EAC significantly improves detection accuracy compared to standard convolutions. 4. Action Insight Explorer (360AIE) with LLM Summarization: We introduce 360AIE, a debriefing interface that summarizes detected actions using large language models (LLMs). This represents the first integration of LLM-based summarization for 360-degree video in mission-critical settings. 5. Optimized Model Deployment: We optimize 360YOWO through quantization and pruning, reducing model size while maintaining accuracy. The optimized model can run on a single consumer-grade GPU, making it accessible to a wider range of users, including firefighter training centers and first responder facilities. II Background - Definitions, and Related Work II-A Definitions Action. An action refers to an observable activity performed by an individual or group in a mission-critical training scenario. Actions range from basic movements (e.g., kneeling) to complex sequences (e.g., deploying a fire hose). The detection granularity depends on the training context and the debriefing requirements. Mission-Critical. A mission-critical scenario involves high-risk operations where delayed or incorrect actions can lead to severe consequences. These include firefighting, emergency response, military operations, and industrial safety, requiring rapid coordination under constrained environments. 360-Degree Video: Opportunities and Challenges. 360-degree video enhances mission-critical training by providing comprehensive situational awareness, eliminating blind spots, and enabling a complete review of trainee actions. It improves training analysis by capturing all interactions for detailed performance evaluations and offers an immersive learning experience, allowing trainees to observe scenarios from multiple

Read the complete text