The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning
Mingkai Tian, Guorong Li, Yuankai Qi, Amin Beheshti, Javen Qinfeng Shi, Anton van den Hengel, Qingming Huang · 2025
Read The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning by Mingkai Tian, Guorong Li, Yuankai Qi, Amin Beheshti, Javen Qinfeng Shi, Anton van den Hengel, Qingming Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23679.txt.
Introduction
Mingkai Tian and six co-authors improve automatic captions for videos that the system was never trained on, in a paper from twenty twenty-five. Such zero-shot captioning leans on vision-language models to suggest text prompts for a language model. Older methods fix on one part of a scene and miss the rest.
Their approach builds three stores of prompts at rising detail: short noun phrases, linked scene graphs, and full sentences. A second mechanism weighs which store entries fit the topic at hand, based on how language is spread around that topic. On three standard benchmarks, the method beat prior systems by clear margins on the main caption score. The authors argue that modeling scene content openly is the key.
Catalog record
| CREATOR | Mingkai Tian, Guorong Li, Yuankai Qi, Amin Beheshti, Javen Qinfeng Shi, Anton van den Hengel, Qingming Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23679
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning Mingkai Tian, Guorong Li, Yuankai Qi, Amin Beheshti, Javen Qinfeng Shi, Anton van den Hengel, Qingming Huang Abstract Zero-shot video captioning requires that a model generate high-quality captions without human-annotated video-text pairs for training. State-of-the-art approaches to the problem leverage CLIP to extract visual-relevant textual prompts to guide language models in generating captions. These methods tend to focus on one key aspect of the scene and build a caption that ignores the rest of the visual input. To address this issue, and generate more accurate and complete captions, we propose a novel progressive multi-granularity textual prompting strategy for zero-shot video captioning. Our approach constructs three distinct memory banks, encompassing noun phrases, scene graphs of noun phrases, and entire sentences. Moreover, we introduce a category-aware retrieval mechanism that models the distribution of natural language surrounding the specific topics in question. Extensive experiments demonstrate the effectiveness of our method with 5.7%, 16.2%, and 3.4% improvements in terms of the main metric CIDEr on MSR-VTT, MSVD, and VATEX benchmarks compared to existing state-of-the-art. 1 Introduction The fact that video captioning remains a challenge, despite significant effort, reflects the inherent complexity of understanding video. Part of the specific problem with captioning is the vast difference between video and natural language as data forms. Video represents a voluminous stream of continuous pixel measurements, whereas natural language is a sequence of discrete symbols with a peculiar structure. Methods that directly model associations between modalities are vulnerable to missing the structure in either. This is visible in the fact that they are susceptible to generating captions that focus on a single scene element. We propose an approach that models the structure of scenes, and the specific language that describes them, at multiple scales in the hope that the resulting model might not be easily distracted. Traditional supervised video captioning methods [ 19 , 32 , 44 , 35 ] utilize an encoder-decoder architecture trained on large-scale, manually annotated video-text pairs. The encoder leverages pre-trained convolutional neural networks ( e.g . , ResNet [ 9 ] , I3D [ 2 ] , C3D [ 8 ] ), while the decoder uses LSTMs [ 11 ] or Transformers [ 15 , 4 ] . Despite achieving remarkable performance, their reliance on human-labeled data constrains real-world scalability. Figure 1 : Current zero-shot captioning methods are easily distracted. NP, SG, and EC denote noun phrase, scene graph (triplets are displayed as concatenated strings), and entire caption prompt, respectively. “Res” indicates the generated captions, with correct and incorrect words highlighted. In the top example, MultiCapCLIP [ 40 ] fails to capture the rider-motorcycle interaction as the prior does not model the distribution of subject-object interactions effectively. In the bottom example, MultiCapCLIP’s top- K retrieval strategy produces repetitive similar noun phrases and lacks person and environment information, because it fails to model either the structure of the scene or of natural language. DeCap [ 16 ] struggles to fully understand the video details due to its coarse-grained prompt of global caption embedding. By contrast, our method generates more accurate and comprehensive descriptions. To address these limitations, zero-shot video captioning has emerged as a promising direction, aiming to generate high-quality descriptions without relying on video-text pairs for training. Existing methods can be broadly categorized into two types: training-free approaches and those trained solely on text corpora. Training-free methods leverage pre-trained vision-language models ( e.g . , CLIP [ 26 ] ) to guide pre-trained language models ( e.g . , GPT-2 [ 25 ] , BERT [ 5 ] ) in generating text during inference. For instance, Tewel et al . [ 31 ] employ randomly initialized pseudo-tokens and prefix prompts ( e.g . , “Video of”) to assist GPT-2 in generating new tokens. After generating a complete sentence, the model updates the pseudo-tokens based on the gradient derived from the CLIP cross-modal similarity. However, introducing visual supervision after token generation can lead to the language model’s priors dominating the captioning process, resulting in hallucinations unrelated to the video content. The other line of work involves training a text decoder on pure text corpora. Textual units ( e.g . , nouns, noun phrases, complete sentences) are extracted from the training corpus to form various memory banks. During training, embeddings of these textual units serve as prefix prompts for the text decoder to reconstruct the original caption. At inference time, visual features are used to retrieve relevant textual semantic units from the memory bank via CLIP similarity, which are then fed into the text decoder. For example, MultiCapCLIP [ 40 ] constructs a memory bank of noun phrases and retrieves top- K elements as prefix prompts. DeCap [ 16 ] , on the other hand, builds a memory bank of entire captions and uses a single token of global embedding as the prefix prompt. Other text-only trained zero-shot image captioning methods, such as MeaCap [ 42 ] , create a memory bank of complete captions as well, retrieve the top- K relevant descriptions, and parse key entities to input into a pre-trained language model. Despite the progress, the above-mentioned methods still face limitations in their prefix prompt construction and retrieval strategies. They often rely on single-granularity textual units, such as nouns, noun phrases, or complete sentences, as prompts, failing to fully exploit multi-granularity textual units to provide rich information for the language model. While noun phrases offer more attribute information than simple nouns, they lack interaction information between entities. In addition, the adopted global embeddings of complete sentences of existing methods may dilute fine-grained details. As illustrated in the top video example of Fig. 1 , MultiCapCLIP fails to accurately capture the rider’s stunt action providing only noun phrase prompts, while DeCap fails to identify the “motorcycle”. Furthermore, top- K retrieval strategies, which simply select the K most similar elements, tend to retrieve semantically repetitive textual units, reducing the diversity of the prompts and the accuracy of the generated captions. We provide such an example in Fig. 1 bottom, where MultiCapCLIP repetitively prompts with musical instrument phrases but neglects information about the person and the environment. To address these challenges, we propose a novel progressive multi-granularity textual prompting strategy. We construct three distinct memory banks comprising noun phrases, scene graphs that incorporate noun phrases, and entire sentences, ensuring the text decoder receives comprehensive semantic cues. Existing parsers [ 17 ] typically extract scene graphs with noun-only nodes, while we propose a text-similarity-based approach to enhance initial scene graphs by incorporating noun phrases with additional attributes wherever possible. We further develop a category-aware retrieval mechanism with top- p filtering for noun phrases and scene graphs, ensuring both diversity and visual relevance. Fig. 1 shows the captions generated by our method. The main contributions of our work are summarized as follows: • We propose a progressive multi-granularity textual prompting strategy, providing the language model with comprehensive semantic information across varying levels of abstraction. • We introduce a category-aware retrieval method with top- p post-processing for semantic units, enhancing the diversity and relevance of the prompts during inference. • Extensive experiments on the MSR-VTT, MSVD, and VATEX benchmarks demonstrate the effectiveness