Boundary-Centric Active Learning for Temporal Action Segmentation

Halil Ismail Helvaci, Sen-ching Samson Cheung · 2026

Read Boundary-Centric Active Learning for Temporal Action Segmentation by Halil Ismail Helvaci, Sen-ching Samson Cheung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15173.txt.

Introduction

Helvaci and Cheung target the costliest part of time-based action segmentation: labeling action boundaries in untrimmed video. Their B-ACT framework spends a fixed clip budget where errors cluster, ranking videos by prediction uncertainty and then picking top boundaries by a new score. That score blends nearby uncertainty, class mix-ups, and time-based prediction shifts to rank each frame's importance. The two-stage loop aims to buy the most accuracy per labeled clip.

Catalog record

CREATORHalil Ismail Helvaci, Sen-ching Samson Cheung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63469
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15173

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boundary-Centric Active Learning for Temporal Action Segmentation Halil Ismail Helvaci, Sen-ching Samson Cheung Abstract Temporal action segmentation (TAS) demands dense temporal supervision, yet most of the annotation cost in untrimmed videos is spent identifying and refining action transitions, where segmentation errors concentrate and small temporal shifts disproportionately degrade segmental metrics. We introduce B-ACT, a clip-budgeted active learning framework that explicitly allocates supervision to these high-leverage boundary regions. B-ACT operates in a hierarchical two-stage loop: (i) it ranks and queries unlabeled videos using predictive uncertainty, and (ii) within each selected video, it detects candidate transitions from the current model predictions and selects the top- K K boundaries via a novel boundary score that fuses neighborhood uncertainty, class ambiguity, and temporal predictive dynamics. Importantly, our annotation protocol requests labels for only the boundary frames while still training on boundary-centered clips to exploit temporal context through the model’s receptive field. Extensive experiments on GTEA, 50Salads, and Breakfast demonstrate that boundary-centric supervision delivers strong label efficiency and consistently surpasses representative TAS active learning baselines and prior state of the art under sparse budgets, with the largest gains on datasets where boundary placement dominates edit and overlap-based F1 scores. I Introduction Understanding human activities automatically in long, unstructured videos is essential for a wide range of applications, including robotics, healthcare, and daily activity recognition [ 23 , 44 , 24 , 11 ] . This has prompted substantial recent research on temporal action segmentation (TAS), whose aim is to assign an action label to each frame, thereby partitioning a long video into consecutive non-overlapping segments. In principle, TAS requires dense temporal supervision—provided either as frame-level labels or segment-level annotations with precise boundary localization [ 11 ] . This process is extremely costly, as untrimmed videos are typically lengthy, requiring annotators to review hours, or even days, of footage and precisely mark action boundaries throughout each video, making data collection difficult to scale. To reduce the cost of annotation in training, several prior studies on TAS have investigated semi-supervised approaches that leverage a limited set of fully annotated videos alongside a larger collection of unlabeled videos [ 58 , 12 , 65 ] . Nevertheless, these methods still depend on fully annotating multiple untrimmed videos, and it remains unclear how to identify the most informative videos for annotation and subsequent training. In contrast, weakly supervised methods rely on coarse annotations for all training videos [ 8 , 9 , 13 , 26 , 34 ] . The performance of both semi-supervised and weakly-supervised approaches are significantly lower than the fully supervised methods. Time-stamp annotation [ 31 , 39 , 42 , 47 ] has achieved closer to fully-supervised performance, however, although only a small number of annotated frames are used for training, they are typically obtained through a costly procedure in which annotators watch every video in its entirety and manually select frames corresponding to distinct actions. A central limitation of these methods is the absence of effective strategies for selecting videos and frames for labeling, which would substantially reduce annotation time and enable a more efficient, guided labeling process. Another strategy for reducing annotation cost is the conventional active learning (AL) paradigm, which iteratively selects unlabeled samples for labeling according to a utility criterion [ 50 ] . Nonetheless, most existing AL studies have centered on sample-level classification, where each instance is associated with a single label. This differs from TAS, in which each video comprises multiple action labels and frame-level labels exhibit strong temporal dependencies. Consequently, both video selection and frame selection must be addressed for efficient labeling. In recent work, hybrid AL for action detection [ 49 , 48 , 28 ] has focused on annotating spatiotemporal regions. However, these methods are limited to short single-action clips spanning only seconds, whereas TAS involves long untrimmed videos with multiple temporally-dependent actions and substantial background content. A key reason TAS does not reduce cleanly to conventional sample-level AL is that the dominant failure modes are not uniformly distributed over frames, but concentrate around temporal transitions. Multiple TAS works explicitly identify boundary ambiguity and the resulting over-segmentation (fragmentation into short sub-segments) as the main obstacles to accurate segmentation in long videos, motivating architectural components and losses that directly target boundaries (boundary-aware cascades, boundary regression, and boundary-smoothing refinements) rather than improving “interior” frames alone [ 66 , 63 , 10 , 64 , 11 , 17 ] . Many methods also emphasize that local continuity should only change at true boundaries, which has specifically motivated researchers to refine segments at the boundaries to reduce boundary ambiguity and over-segmentation [ 11 ] . Even uncertainty-focused modeling in TAS is commonly framed around the ambiguous transition region between actions, further reinforcing that the highest-impact annotation and learning signals lie near boundaries [ 10 ] . Key Contributions. We propose B-ACT (Boundary-centric Active Learning for TAS), a clip-budgeted active learning framework that concentrates supervision on action boundaries, a major source of segmentation error [ 66 , 63 , 10 , 64 , 11 , 17 ] . Our main contributions include: (i) a clip-budgeted annotation protocol that queries only K boundary frames per selected video while leveraging \ell -frame temporal context around each queried boundary, (ii) an uncertainty-guided acquisition policy that selects informative unlabeled videos and prioritizes candidate action transitions using a novel boundary score capturing local confusion, classification ambiguity, and temporal prediction change, and (iii) an efficient two-stage AL loop for TAS that substantially reduces labeling compared to full-video annotation. Extensive experiments on GTEA [ 18 ] , Breakfast [ 33 ] , and 50Salads [ 59 ] demonstrate consistent gains over prior AL baselines. Fig. 1: Comparison of active learning strategies for video. (a) Traditional frame or clip based active learning queries samples within action segments and often requires dense frame-level annotation, resulting in high labeling effort. (b) Proposed boundary-aware active learning prioritizes action boundary frames for querying, requiring labels only at segment transitions and thus reducing annotation cost while still supporting effective model training. II Related Work Temporal action segmentation. TAS has been extensively studied under full supervision, including recurrent models [ 51 , 57 ] , temporal convolutional networks (TCNs) [ 17 , 37 ] , transformers [ 68 , 69 ] , graph neural networks [ 27 , 70 ] , diffusion models [ 41 , 22 ] and hybrid designs that combine convolutional and attention-based backbones [ 43 ] . A dominant line of work employs multi-stage refinement, where predictions are iteratively refined to improve temporal consistency and reduce over-segmentation. Representative methods include MS-TCN [ 17 ] , utilizing temporal convolution networks, ASRF [ 2 ] , decomposing segmentation into action classification and boundary regression [ 2 ] , and ASFormer [ 68 ] , utilizing self-attention [ 61 ] for long-range temporal modeling. To reduce annotation cost, semi-supervised TAS leverages a subset of densely annotated videos (frame-level) [ 12 , 58 , 65 ] , while weakly-supervised TAS relies on coarse supervision such as action transcripts/action sets [ 8 , 9 , 13 , 26 , 34 , 38 ] , sparse frame labels [ 5 , 31 , 39 ] or video-level labels [ 12 , 11 ] . Timestamp-based supervision further

Read the complete text