Prompt2LVideos: Exploring Prompts for Understanding Long-Form Multimodal Videos
Soumya Shamarao Jahagirdar, Jayasree Saha, C V Jawahar · 2025
Read Prompt2LVideos: Exploring Prompts for Understanding Long-Form Multimodal Videos by Soumya Shamarao Jahagirdar, Jayasree Saha, C V Jawahar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08335.txt.
Introduction
Long videos, like full lectures and news broadcasts, are hard for computer models to understand end to end. Soumya Shamarao Jahagirdar, Jayasree Saha, and CV Jawahar assemble a dataset of such long-form videos to study the problem. Labels for hours of footage would need expert annotators, which makes hand-made datasets costly and rare.
Speech recognition can pull the spoken words from a video, and text recognition can read words shown on screen. The authors build baseline systems on those two text streams and show where they fall short. They argue the way forward is prompt engineering: asking large language models the right questions so they can piece a whole video together. Their dataset and baselines give that idea a concrete test bed.
Catalog record
| CREATOR | Soumya Shamarao Jahagirdar, Jayasree Saha, C V Jawahar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08335
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt2LVideos: Exploring Prompts for Understanding Long-Form Multimodal Videos Soumya Shamarao Jahagirdar, Jayasree Saha, C V Jawahar Abstract Learning multimodal video understanding typically relies on datasets comprising video clips paired with manually annotated captions. However, this becomes even more challenging when dealing with long-form videos, lasting from minutes to hours, in educational and news domains due to the need for more annotators with subject expertise. Hence, there arises a need for automated solutions. Recent advancements in Large Language Models (LLMs) promise to capture concise and informative content that allows the comprehension of entire videos by leveraging Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR) technologies. ASR provides textual content from audio, while OCR extracts textual content from specific frames. This paper introduces a dataset comprising long-form lectures and news videos. We present baseline approaches to understand their limitations on this dataset and advocate for exploring prompt engineering techniques to comprehend long-form multimodal video datasets comprehensively. 1 Introduction Learning representations for video-text fusion poses a significant challenge yet holds paramount importance for comprehensive video comprehension across various real-world applications such as video retrieval, captioning, and video based document retrieval analysis. While multimodal contrastive learning leveraging extensive web-scale datasets has demonstrated efficacy in image-text representation tasks, its application in the video-language domain still needs to be explored, particularly in educational and news contexts. A primary hurdle hindering further exploration in this domain is the need for high-quality video-language datasets suitable for large-scale pretraining. Existing research predominantly relies on datasets such as HowTo100M [ 17 ] , MSR-VTT [ 27 ] , YOUCOOK2 [ 29 ] , etc., utilizing automatically generated transcriptions via automatic speech recognition (ASR) or annotated captions. However, these datasets primarily consist of instructional videos focusing on action understanding. The distinct challenges presented by educational and news video content necessitate dedicated attention and dataset curation efforts to facilitate advancements in video-text representation learning within these domains. In education, the focus typically diverges from action understanding, instead emphasizing the transmission of information through mediums like blackboards or PowerPoint presentations. Extracting meaningful insights from these sources poses challenges; solely relying on ASR is insufficient, and acquiring annotations can be complex due to the specialized subject matter knowledge required. Conversely, news videos dynamically convey information through live updates and on-screen graphics. Figure 1: An illustration showcasing the diverse data found within educational and news videos adds intrigue to these domains. Exploring such data holds promise for advancing long video understanding tasks. Consequently, comprehending such content practically requires extracting such live updates to be captured in text form related to the frames within the video. Optical Character Recognition (OCR) may provide useful information in such cases. Existing video-text repositories typically consist of clips ranging from approximately 2 to 10 seconds in length, or they are trimmed to fit within this duration in extensive video datasets. However, this duration may prove ineffective for educational and news videos. Processing clips of uniform size across both domains may be challenging, as educational content often requires extended explanations for specific illustrations, spanning several minutes. Conversely, news videos deliver information swiftly, with updates occurring rapidly. Additionally, special news broadcasts may introduce diverse information with each frame, distinguishing them from the content in the analyzed clips. ASR yields extensive textual data, but it may not be optimal for video-text pre-training due to its overwhelming nature. On the other hand, OCR may need more comprehensive information. Merging these sources could result in the loss of valuable data. Therefore, there is a need for processed information that is tailored for optimal video-text comprehension. Recent advancements in natural language processing and computer vision have led to significant progress in developing large language models (LLMs), profoundly impacting tools within the natural language processing domain. Models like ChatGPT [ 18 ] , LLaVA [ 14 ] , and others can understand and generate human-like text across diverse contexts. Leveraging LLMs to prompt and extract meaningful insights from raw datasets holds immense potential. This study delves into the extraction of additional information through carefully crafted prompts. The extracted insights offer substantial promise for future tasks, including summarization, keyword extraction, identifying key takeaways, and providing valuable metadata that can be repurposed across a broad spectrum of applications tailored to specific downstream tasks. This study aims to pinpoint the prevalent challenges associated with comprehending extended video content, particularly within educational and news contexts Figure 1 . Even though captions accompany individual videos, they often need to catch up in capturing every crucial detail within lengthy videos. For instance, videos labeled with “clustering," focusing on partition clustering, may fail to encompass the entire scope or relevance for those seeking information on subspace clustering techniques. Moreover, there are instances where students may seek particular queries, which are eventually discussed within a video rather than as its primary focus. Likewise, news videos encompass a range of topics, often accompanied by captions highlighting key events or breaking news. However, crucial information may only be captured if it coincides with the highlighted segments or breaking news alerts. In such scenarios, conventional retrieval systems may not yield optimal results. Similarly, videos contain various news, captioned with highlights or breaking news; in such cases, some important information might only be caught if the breaking news or highlights. Therefore, a way to understand long-form videos is needed. ASR and OCR may provide some helpful information for long-form videos. However, the ASR transcripts are often lengthy and must be more concise to capture meaningful captions for each video clip. Conversely, OCR needs help to extract useful information from complex content, such as equations found in specific educational courses. In summary, our contributions are threefold: 1. We investigate the utility of prompt templates for processing text content derived from both ASR and OCR, enabling the utilization of powerful large language models (LLMs) to generate high-quality video-text data with minimal human intervention. 2. To understand the associated challenges in long-form videos, we have curated the Edu-News dataset to enhance multimodal understanding within the domains of education and news through video-language modeling. 3. The analysis of baseline methods for video-text retrieval aims to comprehend the intricacies of the dataset and actively fosters interest in the research community toward this evolving direction. 2 Related Works Video-Language Datasets. Research in the domain of video-language interaction has predominantly concentrated on analyzing movies [ 20 , 24 , 19 ] and instructional content [ 21 , 17 , 1 ] , notably emphasizing instructional videos related to cooking [ 28 , 6 ] . Despite this emphasis, a limited number of studies [ 16 , 7 , 10 , 2 , 3 ] have explored applications within the educational and news domain, which we outline below. Specifically, certain works [ 16 , 7 ] have delved into methods for localizing and recognizing text within educational settings, particularly on blackboards. Others have examined learner engagement with educational videos [ 2 , 3 ] , while another subset of research has focused on segmenting lecture videos [ 10 , 22 ] . News videos frequently present a complex and multifaceted narrative, often depicting multiple events unfolding simultaneously or in quick succession. While there are some datasets tailored for understanding news content, they typically focus on specific