MAMS: Model-Agnostic Module Selection Framework for Video Captioning

Sangho Lee, Il Yong Chun, Hogun Park · 2025

Read MAMS: Model-Agnostic Module Selection Framework for Video Captioning by Sangho Lee, Il Yong Chun, Hogun Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18269.txt.

Introduction

Sangho Lee, Il Yong Chun, and Hogun Park build a chooser that tailors how much of a video an AI captioner should watch, in a 2025 paper. Captioning systems usually grab a fixed number of frames. Too few and key moments vanish; too many and near-duplicate frames waste effort and blur the description.

Their framework, MAMS, works with any captioning model. It first picks a caption module of the right size for the video’s visual tokens, then builds lean subsets of those tokens for each module. The result is frame budgets that fit each video rather than a fixed ration. Better captions for the same compute matter for searchable video archives, accessibility descriptions, and any tool that must narrate footage it has never seen.

Catalog record

CREATORSangho Lee, Il Yong Chun, Hogun Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18269

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAMS: Model-Agnostic Module Selection Framework for Video Captioning Sangho Lee, Il Yong Chun, Hogun Park Abstract Multi-modal transformers are rapidly gaining attention in video captioning tasks. Existing multi-modal video captioning methods typically extract a fixed number of frames, which raises critical challenges. When a limited number of frames are extracted, important frames with essential information for caption generation may be missed. Conversely, extracting an excessive number of frames includes consecutive frames, potentially causing redundancy in visual tokens extracted from consecutive video frames. To extract an appropriate number of frames for each video, this paper proposes the first model-agnostic module selection framework in video captioning that has two main functions: (1) selecting a caption generation module with an appropriate size based on visual tokens extracted from video frames, and (2) constructing subsets of visual tokens for the selected caption generation module. Furthermore, we propose a new adaptive attention masking scheme that enhances attention on important visual tokens. Our experiments on three different benchmark datasets demonstrate that the proposed framework significantly improves the performance of three recent video captioning models. 1 Sungkyunkwan University, Suwon, Republic of Korea 2 Hippo T&C Company Limited, Suwon, Republic of Korea 3 Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea [email protected], {iychun, hogunpark}@skku.edu Introduction The video captioning task generates descriptions for provided videos in natural language ( Li et al. 2021 ; Wang et al. 2019 ) . It has been rapidly gaining attention in blind navigation, video event commentary, human-computer interaction, etc. To improve video captioning performances, it is pivotal to introduce multi-modal transformers ( Sun et al. 2019 ) . Many recent studies extract an identical number of frames for different videos to use a consistent input size for transformer-based models; see, e.g., ( Chen et al. 2023 ; Yang et al. 2023 ) . Figure 1: The overview of the proposed framework. Selecting a fixed number of frames in existing captioning models has critical limitations. For videos with abundant information, e.g., videos with large dynamics, if we extract a limited number of frames, caption generation performances can degrade ( Lin et al. 2022 ) . It may omit frames that encapsulate essential information for caption generation, potentially compromising the accuracy and completeness of caption generation ( Gao et al. 2023 ) . Conversely, for videos with little information, e.g., videos with little dynamics, if frames are densely extracted, it could result in extracting similar frames. Numerous studies have demonstrated that redundant visual tokens generated by a large number of consecutive frames can adversely affect performance ( Liu et al. 2023 ; Liang et al. 2022 ) . Our observations indicate that the performance of existing video captioning methods stagnates or even declines as the number of frames increases. We conjecture that this phenomenon is caused by the fact that existing methods rely on a fixed number of frames across all videos. This study assumes in caption generation that it could be more reasonable to vary the number of frames or visual tokens for each video, rather than to fix it for all videos. To address the aforementioned issues, this paper proposes the first Model-Agnostic Module Selection (MAMS) framework in video captioning that adaptively selects a caption generation module for each video, where each module extracts a different number of frames. The proposed framework consists of an existing caption generator that uses all frames, a smaller caption generator module that uses a subset of frames, and a module & token selector that selects the appropriate generation module and tokens for each video, respectively. Figure 1 illustrates an overview of the proposed framework. The process of the proposed framework is given as follows. (1) We extract visual tokens from video with a video encoder. (2) Using the proposed module & token selectors, we select an appropriate size of a generation module. If a smaller module is selected, we then construct a subset of visual tokens corresponding to selected frames from a full set of visual tokens. (3) We input selected visual tokens – combined with textual tokens – to either a large or small caption generation module. Different from existing models that use a fixed number of frames and visual tokens, the proposed framework adaptively selects a caption generation module with an appropriate size, which results in using a varying number of frames and visual tokens for each video. Moreover, we introduce a new adaptive attention masking scheme that focuses more on visual tokens with higher contributions to caption generation. Ultimately, we select/focus on essential visual tokens in both frame and token levels. The contributions of the paper are summarized as follows: • (Problem discovery) We discover a performance saturation problem in existing captioning methods that extract the same number of frames from all videos. • (New methodology) We propose the first MAMS framework in video captioning that selects an appropriate caption generation module and important visual tokens for each video in terms of frame level. Additionally, we propose a new adaptive attention mask that focuses more on important visual tokens. • (Broad applicability & performance improvement) Our framework is applicable to existing video captioning models and effectively addresses their limitations, significantly improving the captioning performance of the three state-of-the-art models, SwinBERT, UniVL, and mPLUG-2. Notably, applying the MAMS framework to mPLUG-2 achieved a new state-of-the-art benchmark. Related Works Video Captioning Early video captioning approaches used rule-based methods, directly extracting subjects, verbs, and objects to construct sentences ( Das et al. 2013 ; Kojima et al. 2002 ) . Subsequent studies extracted sentences on a frame-by-frame basis and combined them ( Bahdanau, Cho, and Bengio 2015 ; Sutskever et al. 2014 ) . The majority of recent models are based on multi-modal transformers, which simultaneously utilize visual tokens extracted from videos and textual tokens from pre-generated words to generate sentences ( Arnab et al. 2021 ; Sun et al. 2019 ) . Initially, these approaches generated sentences using pre-extracted visual tokens ( Aafaq et al. 2019 ; Pan et al. 2020 ; Pei et al. 2019 ; Shi et al. 2020 ) . Over time, approaches using pre-extracted tokens have advanced into an end-to-end framework that directly extracts visual tokens from raw videos to generate sentences. This end-to-end approach significantly enhances performance, as the visual token extractor can be jointly optimized with the caption generation module ( Lin et al. 2022 ) . We categorize the multi-modal transformer based end-to-end approach into two main classes. The first category is called sparse sampling that selects a limited number of frames from the video ( Fu et al. 2023 ; Wang et al. 2022 ) . This approach could miss important information needed in generating captions for some videos with large dynamics. The second category is called dense sampling that extracts visual tokens with a sufficient number from the consecutive video frames ( Kuo et al. 2023 ; Xu et al. 2023 ; Lin et al. 2022 ) . This approach could generate redundant visual tokens for some videos with small dynamics that negatively affect caption generation performance. This paper proposes a new framework that can overcome the limitations of existing caption generation models using a fixed number of frames. Limitation of a

Read the complete text