Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy

Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton · 2025

Read Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy by Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18509.txt.

Introduction

Faegheh Sardari and three colleagues split a hard video problem into easier pieces. Dense action detection means spotting many actions that overlap in time, and action labels themselves blur together. One network trying to do both jobs struggles.

Their system first detects clear building blocks: the objects involved and the motions performed. Separate sub-networks handle each block, so each one only fights the timing problem. A language-guided training loss also teaches the model which actions tend to occur together. On two tough benchmarks the method beat earlier systems by clear margins.

Catalog record

CREATORFaegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18509

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reframing Dense Action Detection (RefDense): A Paradigm Shift in Problem Solving & a Novel Optimization Strategy Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton Abstract Dense action detection involves detecting multiple co-occurring actions while action classes are often ambiguous and represent overlapping concepts. We argue that handling the dual challenge of temporal and class overlaps is too complex to effectively be tackled by a single network. To address this, we propose to decompose the task of detecting dense ambiguous actions into detecting dense, unambiguous sub-concepts that form the action classes ( i.e. , action entities and action motions), and assigning these sub-tasks to distinct sub-networks. By isolating these unambiguous concepts, the sub-networks can focus exclusively on resolving a single challenge, dense temporal overlaps. Furthermore, simultaneous actions in a video often exhibit interrelationships, and exploiting these relationships can improve the method performance. However, current dense action detection networks fail to effectively learn these relationships due to their reliance on binary cross-entropy optimization, which treats each class independently. To address this limitation, we propose providing explicit supervision on co-occurring concepts during network optimization through a novel language-guided contrastive learning loss. Our extensive experiments demonstrate the superiority of our approach over state-of-the-art methods, achieving substantial improvements of 3.8% and 1.7% on average across all metrics on the challenging benchmark datasets, Charades and MultiTHUMOS. Our code will be released upon paper publication . 1 Introduction Figure 1 : Comparison of current approaches and our proposed approach, RefDense, for tackling the dense action detection task. (a) Current approaches directly address the entire problem ( i.e. , detecting dense, ambiguous actions) using a single network, optimized solely with Binary Cross-Entropy (BCE) loss. In contrast, (b) RefDense decomposes the task into two sub-tasks ( i.e. , detecting dense, unambiguous entity and motion sub-concepts underlying the actions classes) and assigns them to distinct sub-networks. Furthermore, our approach is optimized using both BCE loss and our proposed contrastive co-occurrence language-video loss. Dense action detection aims to recognize and temporally localize all actions within an untrimmed video, even when the actions occur concurrently. A deep understanding of these complex action semantics is crucial for many real-world applications, such as autonomous driving, sports analytics, and complex surveillance, where actions are rarely isolated. To tackle this task, current approaches [ 9 , 24 , 6 , 8 , 21 , 28 ] typically follow a common pipeline. First, the video features are extracted using a pre-trained Encoder (e.g., I3D [ 3 ] , CLIP [ 19 ] ). Then, the features are fed into a Temporal Modeling Block (e.g., multi-scale transformer [ 6 , 21 ] , GNN [ 4 , 8 ] ) to capture temporal relationships among the segments, followed by a classification head that maps the learned representations to multi-action probabilities, enabling dense action detection. Finally, the entire network is optimized using binary cross-entropy (BCE) loss. In dense action detection, beyond the challenge of temporal overlaps, action classes often exhibit semantic overlap ( i.e. , class ambiguity). This overlap can arise from shared entities or motion that define the action classes. For example, in the MultiTHUMOS dataset [ 27 ] , the action classes “Hammer Throw Wind Up” and “Hammer Throw Spin” share an identical entity, a hammer. Similarly, in the Charades dataset [ 22 ] , the classes “Holding a Bag” and “Holding a Sandwich” overlap in motion, the act of holding. We argue that the dual challenge of handling temporal and action class overlaps is too complex to be effectively addressed by the traditional dense action detection pipeline. This motivated us to raise a novel question: Can we reduce the problem’s complexity by eliminating the class overlaps, thereby enabling the network to focus solely on resolving temporal overlaps? To achieve this, we introduce a paradigm shift in solving this task. Instead of directly detecting dense, ambiguous actions using a single network, we propose decomposing the task into detecting dense, unambiguous sub-concepts underlying the action classes ( i.e. , entity and motion sub-concepts), and assigning these sub-tasks to distinct sub-networks. By isolating the unambiguous components of actions, each sub-network can focus exclusively on resolving a single challenge, dense temporal overlaps. To implement this novel paradigm, we (i) design a network comprising two sub-networks, Action-Entity and Action-Motion, and (ii) decompose dense action labels into dense action-entity and dense action-motion labels using prompts and a pre-trained large language model (LLM). While both sub-networks receive the same input video, Action-Entity focuses solely on detecting dense entity concepts involved in dense actions, whereas Action-Motion is dedicated to detecting dense motion concepts involved in dense actions. The dense temporal entity and motion representations learned by the sub-networks are then concatenated for dense action detection. The entire network is optimized using the original dense action labels and the BCE loss, while the Action-Entity and Action-Motion sub-networks are individually optimized using the dense action-entity and dense action-motion labels with the BCE loss. In dense action detection, where multiple concepts can occur simultaneously, awareness of class dependencies can significantly enhance performance. For instance, in scenarios like cooking, actions such as “Pouring” and “Stirring” often occur together. However, we argue that the current dense action detection networks [ 24 , 6 , 21 , 28 ] cannot effectively learn the relationships among the co-occurrence classes as they are trained using the BCE loss which treats each action class independently during the optimization process. This limitation motivates us to raise our second novel question: Can we improve network optimization to fully unlock the potential benefits of co-occurring concepts? To achieve this, we propose providing explicit supervision on co-occurring concepts in the input video during network optimization. Inspired by contrastive language-image pretraining [ 19 ] , we introduce Contrastive Co-occurrence Language-Video learning, which aligns the video features in the embedding space with the textual features of all co-occurring classes. Specifically, we assign a textual description to each co-occurring concepts in the input video and use a frozen pre-trained text encoder to extract their features. Then, we adapt the noise contrastive estimation loss to match the video features with the text features of all co-occurring classes. Through this, the network not only receives explicit knowledge of co-occurring concepts during training, but also implicitly benefits from the learned semantics of related concepts within the embedding space of pre-trained language models. In Fig. 1 , we compare current approaches to our proposed method (RefDense) in tackling the dense action detection task. Our key contributions are summarized as follows: (i) we introduce a paradigm shift in solving dense action detection task—decomposing the problem complexity for the network—an approach that can also benefit solving other dense computer vision problems ( e.g. , dense captioning); (ii) we pioneer the first exploration of explicitly addressing action class ambiguities in dense action detection task; (iii) for the first time, we introduce an optimization process for dense action detection, which enables the network to leverage explicit supervision on co-occurring concepts during training. This approach can enhance the optimization process of any existing or future dense action detection network; (iv) our comprehensive comparison using multiple metrics and challenging benchmark datasets against state-of-the-art approaches demonstrates the superiority of our method, e.g. , achieving substantial improvements of 3.8% and

Read the complete text