Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge

Constantin Patsch, Marsil Zakour, Yuankai Wu, Eckehard Steinbach · 2025

Read Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge by Constantin Patsch, Marsil Zakour, Yuankai Wu, Eckehard Steinbach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06174.txt.

Introduction

Constantin Patsch, Marsil Zakour, Yuankai Wu, and Eckehard Steinbach present a 2025 report on spotting mistakes in first person video for a public challenge. Catching errors as they happen helps in factories and in teaching, since people can fix them at once. Past work mostly caught steps done in the wrong order.

Their system also catches slips in how a step is done, such as a wrong grip or a misused tool. When it flags an error, a large language model writes feedback that explains it. Tests on the HoloAssist benchmark placed their approach second in the mistake detection task. The report gives the design and the results.

Catalog record

CREATORConstantin Patsch, Marsil Zakour, Yuankai Wu, Eckehard Steinbach
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge Constantin Patsch, Marsil Zakour, Yuankai Wu, Eckehard Steinbach Abstract In this report, we address the task of online mistake detection, which is vital in domains like industrial automation and education, where real-time video analysis allows human operators to correct errors as they occur. While previous work focuses on procedural errors involving action order, broader error types must be addressed for real-world use. We introduce an online mistake detection framework that handles both procedural and execution errors (e.g., motor slips or tool misuse). Upon detecting an error, we use a large language model (LLM) to generate explanatory feedback. Experiments on the HoloAssist benchmark confirm the effectiveness of our approach, where our approach is placed second on the mistake detection task. 1 Introduction Action detection and recognition methods Wang et al. 2021 ; Phan et al. 2024 ; Patsch and Steinbach 2023 ; Yang et al. 2022 accurately interpret human actions from video by leveraging spatiotemporal cues. However, a truly intelligent assistant should go beyond recognition and assess the correctness of actions to help users perform tasks accurately. Such systems hold potential for both everyday scenarios like cooking or home maintenance, and industrial applications, such as assembly or mechanical repair. Following recent online mistake detection efforts Lee et al. 2024 ; Seminara et al. 2024 ; Flaborea et al. 2024 , systems should support real-time inference to identify errors from a continuous video stream, enabling users to react promptly and avoid further consequences. An egocentric perspective is particularly beneficial, capturing task execution from the user’s viewpoint and avoiding occlusions common in static views. Recent methods largely focus on procedural errors—like incorrect sequencing or missed actions Lee et al. 2024 ; Seminara et al. 2024 ; Flaborea et al. 2024 —but real-world tasks often involve a broader range of errors. Partial action ordering alone may not fully reflect task success. By also considering execution errors, such as motor mistakes or misuse of tools, a more complete view of task performance emerges. Most current methods further only detect mistakes Flaborea et al. 2024 ; Seminara et al. 2024 without providing insights. However, understanding why an action is wrong, such as missing prerequisites or incorrect execution, can help users correct their behavior. To this end, we leverage recent advances in language and vision-language models Touvron et al. 2023 ; Maaz et al. 2024 ; Li et al. 2023 to generate explanations for detected mistakes using an LLM. Thus, within our approach, we focus on capturing procedural errors that mainly relate to the relative ordering of actions and depend on temporal relations, as well as execution errors, which refer to how the human is performing certain actions. As a result, our approach can better capture the correct execution of a task while being versatile concerning varying error types. Our contribution is three-fold: • We introduce an online mistake-detection method that captures both procedural and execution errors, enabling a more holistic assessment of task execution beyond temporal ordering alone. • Our approach generates explanations for detected mistakes, offering users actionable insights and facilitating error resolution. • Our approach achieves the second place on the Holo-Assist Wang et al. 2023 benchmark 1 1 1 https://www.codabench.org/competitions/2613/ . 2 Methodology In this section, we give an overview of the design of our approach and explain the individual components with respect to the mistake detection and error explanation tasks. Figure 1 : Overview: Our method processes a continuous RGB stream using a video encoder and a Q-Former to extract framewise visual relations. A Video Q-Former captures temporal dependencies between the features, using learnable queries (colored squares) inspired by Li et al. 2023 . These features are fed to the mistake classification layer for final predictions. The Video Q-Former output is gated via g_{m} and projected into the LLM embedding space to generate error explanations. Double slashes indicate layers without backpropagation. 2.1 Mistake Detection As illustrated in Figure 1 , our online setup processes a continuous video stream by dividing the input into frame segments up to the current timestep t . These frames are fed into a visual encoder in combination with a Q-Former, which is based on the architecture introduced in Blip2 Li et al. 2023 , to extract visual features. In our implementation, the video encoder is a Vision Transformer (ViT) Dosovitskiy et al. 2021 . We represent the resulting segment of visual features as v=[v_{t-t_{s}},...,v_{t}] , where v\in\mathbb{R}^{t_{s}\times d_{1}} . Here, t_{s} refers to the number of timesteps within the segment, and d_{1} denotes the dimensionality of the extracted features. Together with learnable queries q\in\mathbb{R}^{t_{q}\times d_{2}} , the feature sequence v is fed into the Video Q-Former. While the standard Q-Former focuses on capturing spatial information within individual frames, the Video Q-Former is designed to model temporal dependencies across multiple frame-wise features. Despite this extension, the Q-Former retains the overall architecture proposed in Li et al. 2023 , similar to the adaptation shown in Zhang et al. 2023 , and is likewise built upon a BERT encoder Devlin et al. 2019 . The output of this process is a set of temporally-aware features denoted as f\in\mathbb{R}^{t_{q}\times d_{2}} . The features f are passed to the mistake classification layer to obtain the final mistake logits m\in\mathbb{R}^{t_{q}} . The error explanation generation process is initiated once a mistake is identified based on those logits. 2.2 Error Explanation The error explanation component aims to offer textual reasoning that clarifies why a particular action is incorrect. These explanations address both execution and procedural mistakes detected within the analyzed segment. The Video Q-Former features f , extracted during the mistake detection phase, are passed through a gating mechanism g_{m} to generate these explanations. This is formally defined as: g_{m}(\sigma(m))=\begin{cases}1,&\text{if }\sigma(m)\geq\tau\\ 0,&\text{otherwise}\end{cases} (1) where \sigma denotes the sigmoid function. When the predicted logit exceeds a predefined threshold \tau , the features from the Video Q-Former are passed to a projection layer. This projection layer is a linear transformation that maps the features into the LLM embedding space. The projected features, along with a prompt, are then input to the LLM to generate the final explanation. This mechanism ensures that explanations are produced only when an error is detected. Holo Assist Methods Modality F1-Score Correct Mistake Prec Rec Prec Rec TSformer (Baseline) RGB 35.1 82.6 51.8 12.9 26.9 TSformer (Baseline) RGB + H(GT) 36.2 85.5 43.1 9.7 11.5 GazeCompl (CVPR24) RGB + E 51.0 95.0 92.0 6.0 9.0 266097 (CVPR25) RGB 54.0 96.0 86.0 9.0 26.0 MR-CAS (CVPR25) RGB 57.0 97.0 60.0 8.0 63.0 Ours RGB 55.0 96.0 91.0 11.0 21.0 Table 1 : Performance comparison on Holo Assist Wang et al. 2023 . TSformer Bertasius et al. 2021 + H(GT) Wang et al. 2023 denotes the TimeSformer model combined with ground truth hand pose as reported by Wang et al. 2023 . E denotes the eye gaze information. Figure 2 : Qualitative results on a video cutout of the HoloAssist dataset, where exemplary frames are sampled from the indicated segments. The confidence scores indicating the mistake detection probabilities \hat{y} and the ground truth indicating the mistake annotations are visualized over time. Holo Assist Methods BLEU ROUGEL CIDEr GT Sampling 0.46 0.50 0.30 Videollama Zhang

Read the complete text