OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan · 2026

Read OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video by Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11102.txt.

Introduction

Pu and three colleagues introduce the video-to-script task and OmniScript, an 8B omni-modal language model for long-form cinematic video. Current multi-modal models handle video grasp well, yet turning long videos into temporally grounded scripts remains hard. The task aims to generate hierarchical scene-wise scripts covering actions, dialogues, expressions, and audio cues. The authors construct a hand-annotated benchmark and propose a time-aware layered evaluation framework. It is trained with reasoning-chain tuning for plot and characters, followed by reinforcement learning with segmented rewards.

Catalog record

CREATORJunfu Pu, Yuxin Chen, Teng Wang, Ying Shan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98419
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan Abstract Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript , an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy. † † affiliation: ARC Lab, Tencent † † contribution: Equal Contribution † † date: April 13, 2026 † † Project Page: http://arcomniscript.github.io † † Code: https://github.com/TencentARC/OmniScript Figure 1 : Overview of our Video-to-Script (V2S) framework. Given a long-form cinematic video, our pipeline performs temporally grounded scene-event parsing and generates a structured script with multimodal fields (dialogue, action, expression, and audio cues). 1 Introduction The analysis of cinematic content is a complex cognitive task, requiring the interpretation of a rich tapestry of visual and linguistic elements, including character relationships, narrative progression, and dialogue nuances. A deep understanding of these components is not only a cornerstone of computational media analysis but also holds significant practical value for the film and television industry, with potential applications in automated logging, content retrieval, and assisting human creativity. While recent advancements in multimodal large language models (MLLMs) [ 19 ; 32 ; 39 ; 7 ] have shown remarkable promise in video understanding, they predominantly focus on short-form video clips and tasks such as captioning or question-answering [ 38 ; 8 ; 26 ; 35 ] . The more ambitious and practical goal of translating a full-length movie or episode into a detailed, structured script remains a largely unexplored frontier. In this paper, we introduce the first Video-to-Script (shown in Fig. 1 ) multimodal large language model, a novel omni system designed to ingest long-form videos, ranging from several to tens of minutes, and generate a comprehensive, scene-by-scene script. This generated output includes not only the visual setting, such as the location, time of day, and atmosphere of each scene, but also a detailed breakdown of the in-scene narrative: character actions, dialogues (including voiceovers), and emotions. This task, which we term “holistic script generation,” pushes the boundaries of current video MLLMs by demanding fine-grained, long-range temporal understanding and coherent narrative synthesis. However, the development of such a model is fraught with significant challenges. First, there is a critical lack of suitable training data. Annotating long-form videos with the necessary granularity, parsing complex multi-scene structures and intricate character interactions, is an exceptionally labor-intensive and time-consuming endeavor. This data scarcity poses a fundamental obstacle to training models capable of deep narrative comprehension. Second, evaluating the quality of generated scripts presents a unique metrological challenge. Unlike tasks with single ground-truth answers, the model’s output is a long, open-ended, and richly structured narrative containing multiple elements with precise temporal stamps. Defining robust metrics that capture the accuracy, coherence, and completeness of such complex generations remains an open problem. Third, the autoregressive generation of such detailed descriptions is computationally prohibitive. Our preliminary analysis indicates that describing a mere two-minute clip requires approximately 4,000 tokens. As the video duration extends, the token count—and consequently the inference cost and time—can explode, creating a critical bottleneck for practical efficiency. To overcome these hurdles, we propose a comprehensive and novel solution consisting of three key contributions. First , we introduce the first-of-its-kind movie and TV series understanding dataset. This dataset encompasses a diverse range of content, including both horizontal-format films and vertical-format short dramas, covering a wide array of themes and genres to ensure robust model training. Second , we establish a dedicated, human-annotated benchmark and a suite of evaluation metrics specifically designed for the video-to-script task. By rigorously assessing the performance of existing closed-source and open-source MLLMs, our benchmark not only quantifies the current state-of-the-art but also highlights the significant gap in research and capability that our work aims to fill. Third , we propose the first omni-modal (audio+visual) script generation model. We leverage extensive audio-visual pre-training, CoT-style SFT with plot and character relationship reasoning, and employ an open-ended reward for script quality. This reward is effectively utilized in a GRPO-based post-training phase to align the model with human preferences for coherent and accurate storytelling. Extensive experiments demonstrate the superiority of our approach. Our model not only achieves performance comparable to state-of-the-art closed-source models, including Gemini 3-Pro, on long-form video script generation tasks. This work represents a significant step toward automating the intricate process of video understanding and narrative generation, opening new avenues for research and application in computational media analysis. 2 Related Work 2.1 Cinematic Video Understanding and Narration Early cinematic understanding evolved from short-clip descriptions [ 24 ; 30 ; 23 ; 27 ] to plot-centric reasoning [ 29 ; 2 ; 17 ] and structural annotations (e.g., AVA [ 14 ] , MovieGraphs [ 31 ] , MovieNet [ 15 ] ). However, a persistent dichotomy remains: macro-plot analyses lack fine-grained temporal grounding, while structural annotations fail to yield readable, coherent narratives. Recent efforts like Movie101 [ 36 ; 37 ] generate role-aware narrations but still predominantly operate on isolated, dialog-free clips, failing to model full-length film intricacies. In contrast, our proposed Holistic Script Generation task requires transcribing full-length videos into temporally anchored, hierarchically structured scripts. Unlike existing datasets [ 24 ; 27 ; 36 ] that entangle multimodal cues into coarse paragraph summaries, our annotation format strictly decouples fine-grained atomic elements, such as individual actions, expressions, dialogues, and audio cues, providing a much more rigorous benchmark for long-term narrative comprehension. 2.2 Dense and Omni-Modal Video Captioning Traditional dense video captioning focuses on localizing salient events but typically yields sparse, concise summaries that neglect rich multimodal nuances [ 16 ; 9 ; 21 ] . Conversely, recent advancements in audio-visual captioning (e.g., AVoCaDO [ 5 ] , video-SALMONN-2 [ 28 ] , and DiaDem [ 6 ] ) achieve deep multimodal integration but generate holistic descriptions devoid of explicit temporal grounding. Empowered by advanced MLLMs, bridging this gap has emerged as a new frontier. Notably, a concurrent work, TimeChat-Captioner [ 34 ] , introduces an ”Omni Dense Captioning” task to generate timestamped, scene-level descriptions. While TimeChat-Captioner pioneers macro-scene structuring, its character actions, intents, and dialogues remain entangled within coarse summaries. Our OmniScript framework transcends these limitations by enforcing a strictly decoupled, hierarchical structure ( Scene \rightarrow Event \rightarrow Field ) that explicitly isolates fine-grained atomic elements anchored to dense timestamps, effectively resolving the ambiguity present in prior captioning paradigms. 3 Video-to-Script Generation In this section, we first define the cinematic video-to-script problem and output schema, then describe the memory-augmented automatic annotation pipeline used to construct high-quality training data. 3.1 Problem Definition Our task

Read the complete text