TI-PREGO : Chain of T hought and I n-Context Learning for Online Mistake Detection in PR ocedural EGO centric Videos
Leonardo Plini, Luca Scofano, Edoardo De Matteis, Guido Maria D’Amely di Melendugno, Alessandro Flaborea, Andrea Sanchietti, Giovanni Maria Farinella, Fabio Galasso, Antonino Furnari · 2024
Read TI-PREGO : Chain of T hought and I n-Context Learning for Online Mistake Detection in PR ocedural EGO centric Videos by Leonardo Plini, Luca Scofano, Edoardo De Matteis, Guido Maria D’Amely di Melendugno, Alessandro Flaborea, Andrea Sanchietti, Giovanni Maria Farinella, Fabio Galasso, Antonino Furnari (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02570.txt.
Description
Abstract Identifying procedural errors online from egocentric videos is a critical yet challenging task across various domains, including manufacturing, healthcare and skill-based training. The nature of such mistakes is inherently open-set, as unforeseen or novel errors may occur, necessitating robust detection systems that do not rely on prior examples of failure.
Catalog record
| CREATOR | Leonardo Plini, Luca Scofano, Edoardo De Matteis, Guido Maria D’Amely di Melendugno, Alessandro Flaborea, Andrea Sanchietti, Giovanni Maria Farinella, Fabio Galasso, Antonino Furnari |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02570
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TI-PREGO : Chain of T hought and I n-Context Learning for Online Mistake Detection in PR ocedural EGO centric Videos Leonardo Plini, Luca Scofano, Edoardo De Matteis, Guido Maria D’Amely di Melendugno, Alessandro Flaborea, Andrea Sanchietti, Giovanni Maria Farinella, Fabio Galasso, Antonino Furnari Abstract Identifying procedural errors online from egocentric videos is a critical yet challenging task across various domains, including manufacturing, healthcare and skill-based training. The nature of such mistakes is inherently open-set, as unforeseen or novel errors may occur, necessitating robust detection systems that do not rely on prior examples of failure. Currently, no existing technique can reliably detect open-set procedural mistakes in an online setting. We propose a dual-branch architecture to address this problem in an online fashion: the recognition branch takes input frames from egocentric video, predicts the current action and aggregates frame-level results into action tokens while the anticipation branch leverages the solid pattern-matching capabilities of Large Language Models (LLMs) to predict action tokens based on previously predicted ones. Mistakes are detected as mismatches between the currently recognized action and the action predicted by the anticipation module. Extensive experiments on two novel procedural datasets demonstrate the challenges and opportunities of leveraging a dual-branch architecture for mistake detection, showcasing the effectiveness of our proposed approach. \equalsenior Co-senior role. \equalsenior Co-senior role. \equalsenior Co-senior role. keywords Egocentric Vision, Procedural Mistake Detection, Video Understanding, Large Language Models, Chain of Thought, In-Context Learning † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. † † equal-contributors: These authors contributed equally to this work. 1 Introduction Detecting procedural errors in videos has recently gained significant attention as it holds the potential to enhance safety and efficiency in several fields. Humans engage in procedural activities everyday in different contexts, such us daily activities (e.g., cooking a meal), hobbies (e.g., fixing a bike), and professional settings (e.g., operating complex machineries). Monitoring procedural activities is crucial to guarantee the desired results and operate safely, especially in high stakes contexts such as industrial workflows. Wearable devices, looking at the scene from the user’s point of view, and endowed with the capability of providing feedback through augmented reality, can be used to assess procedural mistakes (e.g., forgetting to add water to the kettle before turning it on) and provide real-time feedback. This immediate feedback allows for timely corrections, fostering faster skill development, skill acquisition and safer learning environments in high-stakes areas like surgery and industrial workflow. Figure 1: Procedural Mistake Detection involves identifying errors within a procedural video. Each procedure is composed of different steps that should be executed with a certain order. The aim is to develop a method capable of analyzing a video and determining whether each frame contains a mistake, e.g., a step that is not executed in the correct order. Addressing this task is particularly useful when applied to wearable devices, as they allow for direct feedback to the person performing the task. In the Figure, TI-PREGO, depicted by a gear icon, takes as input the video sequence from time 0 to t-1 , classifying the current frame t as either correct ( green ) or a mistake ( red ). This process continues frame by frame until a mistake is detected. Advanced mistake detection models will become essential to ensure precision, safety, and efficiency in procedural applications. This demand has recently resulted in an increasing number of datasets Jang et al. (2019) ; Sener et al. (2022) ; Wang et al. (2023) ; Ghoddoosian et al. (2023) ; Ding et al. (2023) ; Tang et al. (2019) ; Ben-Shabat et al. (2021) ; Schoonbeek et al. (2024) ; Zhukov et al. (2019) ; Elhamifar and Naing (2019) ; Miech et al. (2019) ; Ragusa et al. (2024) and methodologies designed to advance procedure learning Lu and Elhamifar (2022) ; Ghoddoosian et al. (2023) ; Wang et al. (2021b) ; Zhong et al. (2023) ; Jang et al. (2019) ; Tang et al. (2019) ; Seminara et al. (2024) and error detection models Wang et al. (2023) ; Ding et al. (2023) ; Schoonbeek et al. (2024) . However, existing mistake detection approaches vary widely. Some methods emphasize action detection, identifying specific errors like missing steps, out-of-sequence actions Narasimhan et al. (2023) or incorrect ordering Seminara et al. (2024) , while other models bypass action detection altogether and monitor changes to the assembled object to ensure procedural accuracy Chen et al. (2020) ; Candido de Oliveira et al. (2023) . This variability in proposed approaches and their intended objectives highlights the need for more comprehensive evaluation frameworks and standardized benchmarks to assess and compare the effectiveness of different mistake detection strategies and for more olistic models, able to detect any deviation from the correct workflow. An ideal Procedural Mistake Detection (PMD) model should demonstrate two critical capabilities: robustness to diverse mistake types and the ability to provide online, timely feedback. Regarding robustness, a reliable PMD system should be capable of identifying any mistake that may occur within a structured procedure, regardless of its type. To uphold this principle, in this paper we propose framing mistake detection as a One-Class Classification (OCC) problem, where the model is trained to learn the correct sequences of steps within a procedure and is then able to recognize any deviation from the correct workflow. Within the OCC framework, during training, the PMD model is only exposed to “normal” procedures Flaborea et al. (2023) ; Zaheer et al. (2022) to learn the correct workflows, i.e. the sequences of performed steps that allow to correctly complete the task. Conversely, during inference, the learned model should detect any deviation from the learnt workflow, deeming that step as an error. Further, it is worth noting that the OCC framework requires only binary labels ( Correct vs. Mistake ), which are easier to collect with respect to fine-grained labels, such as the error type , when labelling videos of procedural datasets. Second, the model should provide online 1 1 1 We distinguish an online setup, which allows slight processing delays while enabling effective feedback, from a real-time setup, which requires immediate responses. feedback quickly enough to allow users to correct mistakes promptly, minimizing the reinforcement of incorrect actions and reducing the risk of hazardous situations. It is worth noting that the requirement for immediate feedback defines the online setting and imposes a strict causal constraint on the PMD models as they are allowed to leverage only past and present information for producing their decisions. In contrast, previous methods Ghoddoosian et al. (2023) ; Ding et al. (2023) ; Wang et al. (2023) ; Lee et al. (2024) adopt an offline framework, where the analysis is performed using complete pre-recorded procedures. This allows to employ bidirectional models that leverage both past and future context, achieving high accuracy in retrospective error identification for post-hoc analysis like performance grading, but making them unsuitable for live intervention. Thus, we argue that the processing approach must shift towards proactive error anticipation, where the model predicts the next correct step of the sequence to timely flag deviations. Besides these two pivotal abilities, the PMD system should leverage an egocentric perspective in procedural videos, enhancing the model’s applicability