HierSum: A Global and Local Attention Mechanism for Video Summarization
Apoorva Beedu †, Irfan Essa †‡ · 2025
Read HierSum: A Global and Local Attention Mechanism for Video Summarization by Apoorva Beedu †, Irfan Essa †‡ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18689.txt.
Introduction
Apoorva Beedu and Irfan Essa present HierSum, a method that summarizes how-to videos by their steps, in a 2025 paper. A good summary of a lesson video should keep each key step and drop the rest. Their method reads fine detail from subtitles and pairs it with the video-wide instructions. It also learns from a viewing signal: the parts audiences replay most tend to matter.
The method splits videos into meaningful segments that match the steps of the task. On four benchmark sets it beat rival methods on accuracy and ranking scores. The team also built a fresh data set from how-to sites, pairing videos with written steps. Training on that set lifted results further, and removal tests showed which parts carry the gains.
Catalog record
| CREATOR | Apoorva Beedu †, Irfan Essa †‡ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18689
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HierSum: A Global and Local Attention Mechanism for Video Summarization Apoorva Beedu †, Irfan Essa †‡ Abstract Video summarization creates an abridged version (i.e., a summary) that provides a quick overview of the video while retaining pertinent information. In this work, we focus on summarizing instructional videos and propose a method for breaking down a video into meaningful segments, each corresponding to essential steps in the video. We propose HierSum , a hierarchical approach that integrates fine-grained local cues from subtitles with global contextual information provided by video-level instructions. Our approach utilizes the “most replayed” statistic as a supervisory signal to identify critical segments, thereby improving the effectiveness of the summary. We evaluate on benchmark datasets such as TVSum, BLiSS, Mr.HiSum, and the WikiHow test set, and show that HierSum consistently outperforms existing methods in key metrics such as F1-score and rank correlation. We also curate a new multi-modal dataset using WikiHow and EHow videos and associated articles containing step-by-step instructions. Through extensive ablation studies, we demonstrate that training on this dataset significantly enhances summarization on the target datasets. The code and dataset will be released upon acceptance. 1 Introduction Finding the right video content online continues to be a challenge. Simple search queries such as “How to make an omelet?” give hundreds of results, with videos ranging from a couple of minutes to more than ten minutes. This makes it harder to identify relevant videos from the hundreds of suggestions. Extracting and identifying essential and relevant video segments for a given query significantly, i.e., Video Summarization, improves the experience of watching task-related videos. It seeks to automatically select the most critical segments in a video, condensing the content while preserving essential information. Uni-modal techniques such as keyframe extraction [ 5 , 52 ] , scene segmentation [ 11 , 39 ] , and keyframe importance voting [ 48 ] have been explored for summarization. There is also a growing interest for multimodal summarization [ 21 , 57 , 33 ] , which directly find the necessary keyframes by effectively merging information from multiple modalities [ 33 , 37 , 19 ] , or by using text data to align information [ 38 , 21 ] , usually in a single-stage approach. These techniques are developed on datasets like TVSum [ 48 ] and SumMe [ 18 ] , which provide frame-level importance scores. As manually annotating every frame is expensive, these datasets tend to be smaller and include only a limited range of events or actions of interest. Figure 1 : Instructional video on “How to Bake Salmon” from WikiHow showing ASR text and instructions from the associated WikiHow article. Subtitles sometimes miss important instructions such as “Leave the skin on”, whereas global descriptions lack video-text alignment. To address the issue of scale and diversity of tasks in standard datasets, recent methods generate pseudo-summaries. He et al . [ 21 ] extract keyframes by matching frames with video thumbnails. TL;DW [ 38 ] generates pseudo summaries from a large corpus of instructional videos by leveraging consistency of essential steps across different videos of the same task. However, this approach tends to capture only the most frequent steps, overlooking unique yet important actions specific to individual videos [ 38 ] . In contrast, [ 49 ] utilizes replay statistics from YouTube videos to generate scores, with these statistics highlighting key moments; He et al . [ 49 ] further demonstrate that these measures effectively aid video summarization. Although these methods partially mitigate the challenges posed by small-scale datasets, they fail to exploit the detailed instructions that instructional videos provide . To tackle this issue, we propose to use both clip-level subtitles and video-level instructions during training. The essential events in a video are often captured by either Automatic Speech Recognition (ASR)-generated subtitles, video-level descriptions, or combination of both. As illustrated in Figure 1 , the description contains all the relevant information about the task “ How to Bake Salmon ” including additional details such as Leave the skin on , while subtitles provide the instructions in a more fine-grained manner. We show that training a teacher-student model at global and local levels, combined with the most replayed scores, successfully captures all important segments in videos, leading to effective summarization. This approach results in a nearly 2% improvement on the rank coefficient metric for TVSum, 2% improvement on F1-score for Mr.HiSum, and 1% improvement on cosine similarity for the BLiSS dataset. In summary, our contributions are: • We introduce a novel hierarchical framework that seamlessly integrates local features from subtitles with global video-level instructions. Employing parent-child training strategy, our approach effectively captures both long-term contextual information and detailed subtitle-level cues. • Our experiments demonstrate that incorporating highlight labels – derived from most replayed statistics, in conjunction with text inputs significantly enhances video summarization performance. Additionally, ablation studies and extensive experiments indicate that pre-training on the most replayed statistics datasets yields an improvement of approximately 1–2% on target datasets. 2 Related Work Video Summarization can be broadly classified into: (i) Video-to-Video summarization [ 7 , 21 , 48 ] – where the task is to select important frames in the video to create a concise summarized video; and (ii) Video-to-text summarization [ 19 , 22 ] , with the objective of generating a text description that describes all the relevant information in a video in a concise manner. Video-to-Video Summarization (hereinafter referred to as Video summarization, for clarity) approaches are further categorized into unsupervised learning [ 43 , 63 , 3 , 26 , 10 , 38 ] and supervised learning [ 48 , 21 , 2 , 14 , 23 , 40 , 24 , 64 , 8 , 39 ] . Supervised methods use datasets such as TVSum [ 48 ] and SumMe [ 18 ] , which are smaller in size and lacking diversity. In contrast, most unsupervised methods rely on clustering [ 38 , 10 ] techniques, where the frames are clustered based on either frame similarity or video-text alignment scores. IV-Sum [ 38 ] trains on pseudo summaries generated from a large corpus of instructional videos through task relevance scores, and steps that are relevant to the task appear across many videos. Mr.HiSum [ 49 ] , on the other hand, uses the most replayed statistics from YouTube for training. While these stats are synonymous to highlights, Sul et al. show that it transfers well to video summarization tasks via a zero-shot setting. Our proposed work builds on this approach and uses the most-replayed statistics for supervision, in addition to using the available video-level information and step-level instruction data during training. Multi-Modal Summarization aims at using multiple modalities to provide additional context to the model, leading to improved summarization. Existing works [ 7 , 29 , 28 , 54 ] utilize complementary information from optical flow, audio, etc., to enhance summarization. In particular, language-guided frameworks have seen increasing interest in recent literature [ 37 , 21 , 38 , 28 , 41 ] . To enable such video-text learning for summarization, VideoXum [ 32 , 19 , 55 ] and UBiSS [ 35 ] (building on ActivityNet and QVH datasets), created datasets for Video-Video and Video-Text summarization.