Towards an Automated Multimodal Approach for Video Summarization: Building a Bridge Between Text, Audio and Facial Cue-Based Summarization

Md Moinul Islam, Sofoklis Kakouros, Janne Heikkilä, Mourad Oussalah · 2025

Read Towards an Automated Multimodal Approach for Video Summarization: Building a Bridge Between Text, Audio and Facial Cue-Based Summarization by Md Moinul Islam, Sofoklis Kakouros, Janne Heikkilä, Mourad Oussalah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23714.txt.

Introduction

Md Moinul Islam, Sofoklis Kakouros, Janne Heikkilä, and Mourad Oussalah built a video summary method in two thousand twenty-five that uses more than words alone. Growing piles of video in school, work, and social life need short forms that keep the key points. Their method joins text, sound, and face cues to pick the moments that matter most.

Words stressed across more than one channel, which they call bonus words, help flag key moments. Summaries are lined up with time stamps in the video. In tests, their method beat a classic text-only method by a wide margin on both text and video scores.

Catalog record

CREATORMd Moinul Islam, Sofoklis Kakouros, Janne Heikkilä, Mourad Oussalah
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-23714

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards an Automated Multimodal Approach for Video Summarization: Building a Bridge Between Text, Audio and Facial Cue-Based Summarization Md Moinul Islam, Sofoklis Kakouros, Janne Heikkilä, Mourad Oussalah Abstract The increasing volume of video content in educational, professional, and social domains necessitates effective summarization techniques that go beyond traditional unimodal approaches. This paper proposes a behaviour-aware multimodal video summarization framework that integrates textual, audio, and visual cues to generate timestamp-aligned summaries. By extracting prosodic features, textual cues and visual indicators, the framework identifies semantically and emotionally important moments. A key contribution is the identification of bonus words , which are terms emphasized across multiple modalities and used to improve the semantic relevance and expressive clarity of the summaries. The approach is evaluated against pseudo-ground truth (pGT) summaries generated using LLM-based extractive method. Experimental results demonstrate significant improvements over traditional extractive method, such as the Edmundson method, in both text and video-based evaluation metrics. Text-based metrics show ROUGE-1 increasing from 0.4769 to 0.7929 and BERTScore from 0.9152 to 0.9536, while in video-based evaluation, our proposed framework improves F1-Score by almost 23%. The findings underscore the potential of multimodal integration in producing comprehensive and behaviourally informed video summaries. [[email protected], ] *1 [[email protected], ] keywords multimodal video summarization ,computer vision ,natural language processing ,audio and speech processing ,multimedia content creation † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † address: Center for Machine Vision and Signal Analysis, Faculty of ITEE, University of Oulu, Finland † † corresp: Corresponding author. † † venue: HHAI-WS 2025: Workshops at the Fourth International Conference on Hybrid Human-Artificial Intelligence (HHAI), June 9–13, 2025, Pisa, Italy 1 Introduction In recent years, the consumption of video content has reached unprecedented levels, becoming a central part of modern communication, education, and professional interactions. Platforms hosting interviews, lectures, and virtual meetings have become essential tools for sharing knowledge and fostering collaboration. Unlike static forms of media such as text or images, video integrates spoken language, facial expressions, and vocal tone, providing a multi-sensory experience that enhances the richness of the conveyed information. However, this abundance of video content brings with it a challenge: information overload. Viewers often face the difficult task of navigating through long videos to extract key insights, a process that becomes increasingly impractical as the volume of content continues to grow. This has led to the development of automatic video summarization systems, which aim to condense long videos into concise summaries that retain the core messages and important context Apostolidis et al. 2021 . Traditional video summarization methods have been largely based on unimodal techniques. Visual-based methods, such as those that detect scene boundaries or extract key-frames, are effective in analyzing the structural aspects of the video. However, these approaches often miss the deeper meaning conveyed through audio or behavioral cues Chu et al. 2015 . On the other hand, text-based methods, which summarize transcribed speech, capture linguistic content but fail to consider the emotional and contextual nuances provided by visual and auditory features, such as tone of voice or facial expressions Tiwari and Bhatnagar 2021 . These single modality-based approaches struggle to fully represent the richness of human communication, where meaning often arises from the integration of multiple sensory elements Otani et al. 2017 . To address these limitations, multimodal video summarization has emerged as a promising solution. By combining visual, auditory, and textual cues, multimodal methods can provide a more comprehensive understanding of content. Visual cues, such as facial expressions and body posture, provide insight into the speaker’s emotions and communicative intent, while auditory features, such as pitch and loudness, highlight emotional states. Textual data, aligned with the video timeline, provides semantic context, enabling a more accurate and contextually rich summary Rochan et al. 2018 . In this work, we propose a novel multimodal video summarization framework that integrates audio, visual, and textual cues to identify important video segments. The framework focuses on identifying significant changes in behavior, such as variations in vocal intensity or notable gestures, and correlating these with spoken content to produce more relevant summaries. One key feature of our approach is the identification of bonus words , which are terms that are emphasized through audio or visual cues. These bonus words are used to enhance the relevance of the summary by highlighting words that carry additional semantic or emotional weight. Our approach generates both a concise textual summary and a timestamp-aligned video summary, ensuring that the audiovisual context is preserved. To direct our research, we propose the following research questions relevant to our proposed work. (RQ1) How can multimodal cues, such as audio prosody, visual expressions, and textual alignment, be effectively used to identify important events in video content? (RQ2) How does the inclusion of bonus words , emphasized through visual or auditory cues, impact the informativeness and relevance of generated summaries? (RQ3) How does aligning transcript data with timestamped prosodic and visual cues improve the coherence and precision of both video and text-based summaries? This work presents a novel approach to multimodal video summarization. Our contribution focuses on the integration of multimodal analysis, extending beyond static content extraction to capture audio, text and visual cues. The main contributions are as follows: • We propose a novel multimodal summarization framework that combines audio (prosodic features), visual (expressions and gestures), and textual (transcripts) cues to create timestamp-aligned textual and video summaries, addressing the limitations of traditional unimodal methods. • We introduce a methodology for identifying bonus words , terms that are emphasized through visual, audio, and textual cues. By prioritizing these words, we improve the emotional expressiveness and semantic accuracy of the generated summaries. • We evaluate the generated summaries using both text-based and video-based metrics, providing a more comprehensive framework for assessing the quality and relevance of multimodal video summaries. 2 Related Works Text Summarization. Text summarization addresses the increasing demand for condensing large volumes of digital text into concise and meaningful summaries. Among the various summarization techniques, extractive summarization is particularly relevant to this work. This method selects key phrases directly from the original text, ensuring factual accuracy, but may struggle with maintaining smooth transitions and coherence between ideas Wibawa et al. 2024 . A widely used approach within extractive summarization is the Edmundson heuristic method, which scores sentences based on features such as term frequency, sentence position, and cue phrases, allowing for the selection of the most important sentences Edmundson 1969 . This technique is practical, systematic, and well-suited for the methodology employed in this study. Although abstractive summarization, which generates new sentences for better fluency and readability, exists as an alternative, it is less relevant to the current focus and can sometimes introduce factual inaccuracies Wibawa et al. 2024 ; Islam et al. 2024 ; See et al. 2017 . While recent advancements, particularly transformer models, have enhanced the effectiveness of both extractive and abstractive summarization, the Edmundson heuristic approach remains a reliable and effective foundation for extracting key information in this context. Video Summarization. Traditional video summarization methods have primarily focused on extracting key frames or segments that represent the core content of the video. With the rise of machine learning

Read the complete text