HOI-aware Adaptive Network for Weakly-supervised Action Segmentation

Runzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, Yap-Peng Tan · 2026

Read HOI-aware Adaptive Network for Weakly-supervised Action Segmentation by Runzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, Yap-Peng Tan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26227.txt.

Introduction

Zhang and five colleagues built AdaAct, a network that segments actions in video under weak supervision, meaning only rough labels guide training. Long-range human-object interaction patterns supply context that separates look-alike actions such as pouring juice from pouring coffee. A video encoder distills the most telling interactions, and a two-branch HyperNetwork retunes the temporal encoder for each video on the fly.

Catalog record

CREATORRunzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, Yap-Peng Tan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31602
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.26227

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HOI-aware Adaptive Network for Weakly-supervised Action Segmentation Runzhong Zhang, Suchen Wang, Yueqi Duan, Yansong Tang, Yue Zhang, Yap-Peng Tan Abstract In this paper, we propose an HOI-aware adaptive network named AdaAct for weakly-supervised action segmentation. Most existing methods learn a fixed network to predict the action of each frame with the neighboring frames. However, this would result in ambiguity when estimating similar actions, such as pouring juice and pouring coffee. To address this, we aim to exploit temporally global but spatially local human-object interactions (HOI) as video-level prior knowledge for action segmentation. The long-term HOI sequence provides crucial contextual information to distinguish ambiguous actions, where our network dynamically adapts to the given HOI sequence at test time. More specifically, we first design a video HOI encoder that extracts, selects, and integrates the most representative HOI throughout the video. Then, we propose a two-branch HyperNetwork to learn an adaptive temporal encoder, which automatically adjusts the parameters based on the HOI information of various videos on the fly. Extensive experiments on two widely-used datasets including Breakfast and 50Salads demonstrate the effectiveness of our method under different evaluation metrics. 1 Introduction Figure 1: (a) Most existing methods estimate the action probability of frame t using features of the adjacent frames (blue box in the figure). They are difficult to distinguish actions such as pouring juice, coffee, and water considering the high representation similarity. (b) Our method exploits temporally global but spatially local HOI information to learn an adaptive temporal encoder, which provides essential contextual information to distinguish similar actions. As shown in the figure, considering the interactions with a knife, orange, and squeezer in the video, the query action would be more likely to be pouring juice rather than coffee or water. Action segmentation aims to predict the action for every frame in the video. While previous methods have achieved remarkable performance in the fully-supervised setting 17 ; 19 ; 35 ; 37 ; 43 ; 30 , framewise annotation still requires huge labor costs and is hard to obtain. Therefore, action segmentation with weaker forms of supervision gradually gains its popularity in recent years. In particular, transcript supervision 2 ; 18 ; 12 ; 6 ; 21 ; 28 provides an ordered list of actions occurring in the video without the starting and ending time, which significantly reduces the annotation costs and improves the applicability to a rapidly-growing number of videos on the Internet. To learn from the transcript, previous approaches mainly follow the “generating-matching” pipeline 34 ; 21 ; 28 . With the given training videos, they first apply a temporal encoder to generate framewise action probabilities, and then match the predicted probabilities sequence with the transcript based on Viterbi decoding or dynamic time warping. However, to estimate the action probability of frame t during the generating step, most existing approaches only take a fixed number of neighbor frames around it 34 ; 21 , and feed such video clip features into an RNN-based 5 architecture. In this case, the temporal encoder would fail to distinguish the attribute of similar actions such as pouring coffee and pouring juice, which may lead to counter-intuitive results of pouring coffee in a juice-making video. Although tremendous efforts have been made to remedy such ambiguity in the matching step, the results are still unsatisfying due to the inherent defect in the previous generating process. In this paper, we address the ambiguity problem by designing an adaptive weakly-supervised action segmentation framework called AdaAct. Different from previous methods which take a series of fixed-length video clips as input successively (as shown in Figure 1 (a)), we exploit rich contextual information from temporally global but spatially local human-object interactions (HOI) throughout the whole video. Such HOI sequence further instructs the network as prior knowledge, where our temporal encoder can be dynamically adapted to it at the test time. As illustrated in Figure 1 (b), our method first extracts key interactions with objects at different video timestamps, such as the knife, orange, and squeezer. The obtained HOI sequence is further incorporated into the temporal encoder, thereby the network parameters of the encoder dynamically change with the HOI information on the fly. More specifically, we design a three-step video HOI encoder with the “extracting-selecting-integrating” process. We first apply a pre-trained HOI detector to extract positive interaction bounding boxes for the whole video and design a simple selecting algorithm to pick the most representative ones from them. Then, we explore the relations between these key HOI boxes and integrate them into a single feature vector via a transformer-based network. To dynamically adapt the network parameters, we propose a two-branch HyperNetwork that simultaneously learns HOI-dependent and HOI-independent knowledge. Our HOI-independent branch aims to unearth the general characteristics of instructional videos by iteratively updating a learnable embedding list throughout the training process. Such transferable information will later be encoded as a part of instruction used in the temporal encoder during the test time. On the other hand, our HOI-dependent branch takes the encoded feature vector from the video HOI detector as input and adapts the temporal encoder to the given HOI knowledge occurring in the video. Finally, late fusion is utilized to merge the knowledge from two branches, resulting in more precise action segmentation. We summarize our key contributions as follows: 1) To our best knowledge, this is the first work to learn an adaptive temporal encoder for weakly-supervised action segmentation, where the parameters of the network are dynamically adapted according to the input video on the fly. 2) We propose to exploit temporally global but spatially local HOI information in weakly-supervised action segmentation, which provides essential contextual information to address the ambiguity problem of similar actions. 3) We validate our method on two challenging datasets, Breakfast and 50Salads, and achieve state-of-the-art results for both weakly-supervised action segmentation and alignment tasks. 2 Related Work Figure 2: Overview of the network architecture. Our method simultaneously learns HOI-dependent knowledge s from the video HOI encoder and HOI-independent knowledge z from various videos across the dataset. The obtained knowledge is further incorporated through the two-branch HyperNetwork and late fusion, which generates the network parameters of the adaptive temporal encoder. In this way, our network dynamically adapts to the video contents when estimating the action probability of frame t , leading to better discrimination for similar actions. Fully-supervised action segmentation. Fully-supervised action segmentation methods learn action segments under the guidance of framewise annotations. Earlier attempts 14 ; 35 applied action classification on the sliding window, followed by non-maximum suppression to filter out redundant predictions. However, these approaches failed to model the temporal dependency between action sequences. Kuehne et al . 17 tackled this problem via the hidden Markov model, while Pirsiavash et al . 31 applied context-free grammar to capture temporal structure. In recent years, various network architectures were proposed for learning long-range dependency. Lea et al . 19 introduced an encoder-decoder architecture for action segmentation and detection. Lei et al . 20 further applied deformable convolutions and residual stream. Farruha et al . 8 and Li et al . 24 introduced dilated temporal convolution and proposed a multi-stage temporal convolutional network, while various methods improved the multi-stage network using graph-based temporal reasoning 13 or boundary-aware cascade

Read the complete text