EgoAction: Egocentric Action Composition with Reliability-Aware Temporal Fusion for the EPIC-KITCHENS Action Detection Challenge at CVPR 2026

Zhiheng Fu Zixu Li Zhiwei Chen Fangxu Liu Yupeng Hu Weili Guan Liqiang Nie · 2026

Read EgoAction: Egocentric Action Composition with Reliability-Aware Temporal Fusion for the EPIC-KITCHENS Action Detection Challenge at CVPR 2026 by Zhiheng Fu Zixu Li Zhiwei Chen Fangxu Liu Yupeng Hu Weili Guan Liqiang Nie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24496.txt.

Introduction

Zhiheng Fu and colleagues built EgoAction for a 2026 challenge asking models to locate actions in long first-person kitchen videos and label verb and noun. Their pipeline trains separate verb and noun detectors, then pairs top candidates into action guesses. Verb scores falter at motion changes while noun scores falter amid cluttered hands and objects, so a fixed average of their boundaries can mislead. The team instead weights each boundary by the stronger stream's confidence, steadying localization.

Catalog record

CREATORZhiheng Fu Zixu Li Zhiwei Chen Fangxu Liu Yupeng Hu Weili Guan Liqiang Nie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24923
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24496

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EgoAction: Egocentric Action Composition with Reliability-Aware Temporal Fusion for the EPIC-KITCHENS Action Detection Challenge at CVPR 2026 Zhiheng Fu Zixu Li Zhiwei Chen Fangxu Liu Yupeng Hu Weili Guan Liqiang Nie Abstract The EPIC-KITCHENS-100 Action Detection challenge evaluates whether a model can localize the start and end of each action in long untrimmed egocentric videos and assign the corresponding verb–noun action label. In this report, we formulate our submission as EgoAction ( Ego centric A ction c omposition with reliability-aware t emporal fus ion ), a unified decoupled detection and fusion pipeline. The pipeline uses EPIC-finetuned VideoMAE-L features, trains separate noun and verb temporal detectors with causal temporal modeling, composes action hypotheses from top noun–verb pairs, and introduces a confidence-adaptive boundary fusion rule at post-processing time. The key observation is that verb and noun streams often fail differently: verb scores are sensitive to motion transitions, whereas noun scores are sensitive to hand-object visibility and object clutter. A fixed arithmetic mean of their predicted boundaries can therefore amplify localization errors when one stream degenerates. We replace this hard-coded mean with Dynamic Weighted Fusion (DWF), which normalizes the maximum noun and verb classification confidences into proposal-wise boundary weights and linearly combines the two intervals. This lightweight tensor-only operator shifts boundary authority toward the more reliable stream while preserving the decoupled action scoring mechanism. Together with sliding-window inference, top- K K noun–verb action composition, and class-wise Soft-NMS, EgoAction provides a compact and reproducible system for egocentric temporal action detection. 1 Introduction Recent progress in language-conditioned visual reasoning has shown that fine-grained visual understanding often depends on compositional reasoning 2 ; 19 ; 29 ; 6 ; 32 ; 14 ; 8 ; 4 ; 7 . EPIC-KITCHENS-100 provides a large-scale benchmark for fine-grained, ego-centric video understanding, containing unscripted kitchen activities recorded by head-mounted cameras across 45 kitchens, 100 hours of video, about 20M frames, 90K action segments, and a vocabulary of 97 verb classes and 300 noun classes 11 ; 12 ; 21 . The Action Detection track asks participants to detect every action instance in an untrimmed video and assign a compositional label (v,n) , where v is a verb and n is a noun. The official metric is mean Average Precision (mAP) at temporal IoU thresholds from 0.1 to 0.5, reported for verb, noun, and action labels. Consequently, advancing research on such a comprehensive egocentric benchmark holds great potential to facilitate downstream applications in related fields 18 , including compositional reasoning and retrieval 20 ; 13 ; 1 ; 17 ; 15 ; 30 ; 24 , video understanding 22 ; 5 ; 9 ; 34 ; 10 ; 16 , and multimodal learning 25 ; 33 ; 31 ; 28 ; 3 ; 23 . Unlike trimmed action recognition, this track evaluates the full detection pipeline: a model must search through long untrimmed egocentric recordings, find when each interaction starts and ends, and then assign a valid verb–noun pair. The difficulty therefore does not come from recognition alone, but from the coupling between temporal localization and compositional semantics. A temporally accurate proposal with the wrong noun is counted as wrong; a semantically correct verb–noun pair with a shifted boundary is also penalized, especially at higher tIoU thresholds. This creates a setting where factor-level evidence must be preserved, but the final action prediction must still be fused into a single temporally precise detection. This observation motivates a decoupled but reliability-aware design. Following the strong OpenTAD family of temporal detectors 26 ; 27 ; 32 ; 15 ; 5 , we build on feature-based causal temporal modeling, but adapt it to the 2026 challenge under the available EPIC-finetuned VideoMAE-L features 29 . The original CausalTAD 26 report shows that temporal causality is crucial for TAD and reports strong EPIC-KITCHENS action detection results with InternVideo2 features. In our setting, however, the model ought to operate with separate VideoMAE-L noun and verb features and then compose them into action detections. This exposes three concrete challenges: C1: Boundary ambiguity under egocentric motion. Short interaction phases and head-mounted camera shake create temporally shifted proposals. Even if noun and verb classifiers are correct, their regression heads may localize slightly different start/end points. C2: Stream-specific degeneration. The noun stream can become uncertain when the target object is small, occluded, or confused with kitchen clutter; the verb stream can become uncertain when the motion is subtle or temporally delayed. Directly averaging their coordinates assumes equal reliability and can pull a correct boundary toward a noisy one. C3: Compositional action sparsity. EPIC action labels are formed from 97 verbs and 300 nouns, but only a subset of combinations is common. The detector must maintain rich top- K candidates while avoiding a combinatorial explosion before Non-Maximum Suppression (NMS), the post-processing step that removes duplicate temporal proposals with highly overlapping intervals. To address these challenges, we propose EgoAction ( Ego centric A ction c omposition with reliability-aware t emporal fus ion ), a unified decoupled detection and fusion pipeline. Instead of forcing nouns and verbs into a single joint detector or applying a fixed post-processing average, EgoAction treats the two semantic factors as complementary streams with different localization reliability. The key idea is to preserve noun–verb factorization through VideoMAE-L evidence extraction and causal temporal detection, and to perform action-level coupling only after each stream has produced proposal-level confidence and boundary estimates. This design allows the model to inherit the temporal modeling strength of CausalTAD while explicitly correcting the failure mode that matters for compositional action detection: when one stream is confident and the other is temporally shifted, the final boundary should not give them equal authority. Our contributions are threefold, summarized as follows: • We analyze EPIC-KITCHENS Action Detection as a coupled localization–composition problem and identify three bottlenecks—egocentric boundary ambiguity, stream-specific degeneration, and compositional action sparsity—that explain why direct joint detection or fixed two-stream fusion is insufficient. • We propose EgoAction ( Ego centric A ction c omposition with reliability-aware t emporal fus ion ), a decoupled causal detection pipeline using EPIC-finetuned VideoMAE-L noun and verb features, factor-specific temporal detectors, top- K action composition, and confidence-adaptive Dynamic Weighted Fusion for boundary post-processing. • We achieve 25.94 average action mAP, while maintaining balanced factor-level performance of 28.66 verb mAP and 28.61 noun mAP on the official 2026 EPIC-KITCHENS Action Detection leaderboard. 2 Methodology As shown in Fig. 1 , EgoAction ( Ego centric A ction c omposition with reliability-aware t emporal fus ion ) is a unified feature-based temporal action detection pipeline built on EPIC-finetuned VideoMAE-L features and CausalTAD-style temporal detectors. The core design is to transform an untrimmed egocentric video into two factor-level proposal streams, one for nouns and one for verbs, and then compose them into action detections only after each stream has produced both semantic confidence and temporal boundaries. This avoids treating the 97 \times 300 action space as a monolithic class set and, more importantly, allows the final temporal boundary to be assigned according to stream reliability rather than a fixed average. In the following, we first formulate the action detection problem and notation, and then elaborate each component of EgoAction. 2.1 Problem Formulation Given an untrimmed egocentric video \mathcal{V} , temporal action detection predicts a set \mathcal{Y}=\{(T_{i}^{s},T_{i}^{e},V_{i},N_{i},S_{i})\}_{i=1}^{N}, (1)

Read the complete text