Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation

Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Eshika Khandelwal, Gül Varol, Weidi Xie, Andrew Zisserman · 2025

Read Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation by Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Eshika Khandelwal, Gül Varol, Weidi Xie, Andrew Zisserman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01020.txt.

Introduction

Junyu Xie and seven co-authors generated audio descriptions for films by thinking in shots in a 2025 paper. Audio description narrates the visual track for blind viewers, and edited video is built from shots, so the team made shots the unit of work. Context from neighboring shots and film grammar cues like shot scale guide the writing.

The framework wraps around any vision-language model, open or commercial, with add-on expert modules and no retraining. It beat all past training-free methods and even some fine-tuned ones on several benchmarks. A new action score grades how well motion is conveyed. A second protocol asks systems to draft several candidate descriptions for a human to pick, treating them as assistants.

Catalog record

CREATORJunyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Eshika Khandelwal, Gül Varol, Weidi Xie, Andrew Zisserman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01020

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, Eshika Khandelwal, Gül Varol, Weidi Xie, Andrew Zisserman Abstract Our objective is the automatic generation of Audio Descriptions (ADs) for edited video material, such as movies and TV series. To achieve this, we propose a two-stage framework that leverages “shots” as the fundamental units of video understanding. This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures, to guide AD generation. Our method is compatible with both open-source and proprietary Visual-Language Models (VLMs), integrating expert knowledge from add-on modules without requiring additional training of the VLMs. We achieve state-of-the-art performance among all prior training-free approaches and even surpass fine-tuned methods on several benchmarks. To evaluate the quality of predicted ADs, we introduce a new evaluation measure – an action score – specifically targeted to assessing this important aspect of AD. Additionally, we propose a novel evaluation protocol that treats automatic frameworks as AD generation assistants and asks them to generate multiple candidate ADs for selection. Figure 1 : Overview of our training-free Audio Description (AD) generation framework. The input video clip (corresponding to the AD interval) is extended to include adjacent shots, providing richer temporal visual context. Each video frame is labelled with its corresponding shot number. The extended clip then undergoes a film grammar prediction process, where the thread structure and shot scales are estimated. The AD is generated in two stages: Stage I utilises the predicted cinematic information as training-free prompt guidance to produce dense video descriptions. Stage II then employs an LLM to generate a summarised AD output. 1 Introduction In movies and TV series, Audio Descriptions (ADs) are narrations provided for the visually impaired, conveying visual information to complement the original soundtrack. Their purpose is to ensure a continuous and coherent narrative flow, enabling audiences to follow the plot effectively. Unlike video captions, ADs are constrained by length, prioritising the most visually salient and story-centric information, such as character dynamics and significant objects, whilst omitting redundant details like background figures or unchanging locations. Additionally, ADs are typically produced by professional narrators in a specific style and format, ensuring coherence while not interfering with the original audio. With the advent of Visual-Language Models (VLMs), there has been a growth of interest in automatically generating ADs for both movies [ 97 , 101 , 26 , 27 , 28 , 82 , 50 , 19 ] and TV material [ 92 , 22 ] . However, as anyone who has ever watched a movie or read a book about film editing knows – the fundamental unit of edited video material is the shot , not the frame [ 56 , 40 , 58 ] . Shots are used to structure the video material, defining the granularity and temporal context through choices in its scale (close-up, long shot, etc.) and its movement (panning, tracking, etc.). The “film grammar” is then used to convey meaning through specific editing choices on shot transitions (cuts, fades, etc.), durations, and composition (threads, montage, etc.). Current approaches to automating AD generation, and video large language models (VideoLLMs) in general, are not shot-aware, hindering interpretation of edited video material that has frequent shot transitions [ 76 ] . In this paper, our principal objective is to incorporate shot information and editing structure into the AD generation process. To this end, we consider temporal context in terms of shots, and take account of the two key properties of thread structure and shot scale . Thread structure identifies the sequence of shots captured with the same camera. An example of a scene with multiple simultaneous threads in space-time is shown in Fig. 3 . We develop a robust thread clustering method and use its predictions to guide VLMs in understanding shot-wise relationships. Shot scale typically implies the type of content of the frame. For instance, close-up shots often highlight characters’ facial expressions or gaze interactions, while long shots tend to depict the overall environment and ambience of the scene, as shown in Fig. 4 (top). We leverage this property by developing an off-the-shelf shot scale classifier and employing a scale-dependent prompting strategy to improve the contextual relevance of VLM-generated descriptions. In this paper, we develop a training-free AD generation approach. Inspired by AutoAD-Zero [ 92 ] , as illustrated in Fig. 1 , we adopt a two-stage pipeline in which a VideoLLM generates dense text descriptions in the first stage, followed by an LLM that produces the final AD outputs from this text. We improve the effectiveness of training-free approaches by incorporating two key improvements based on: (i) shot-based temporal context; and (ii) the shot scale and thread structure, that are both crucial to cinematic composition and understanding. Previous methods could describe human-object interactions or human-human interactions, such as looks [ 52 ] , within a frame or shot. With the improvements we introduce, the generated AD also includes such interactions when they are implied by the shot structure. As well as the challenge of generating ADs, another challenge is how to evaluate the predicted ADs. Apart from conventional metrics [ 81 , 48 , 10 , 62 ] , several new AD metrics [ 101 , 28 , 26 ] have been proposed, including CRITIC [ 28 ] measuring who is mentioned in the AD. However, these metrics fail to emphasise character actions – one of the most critical aspects of ADs. To address this, we introduce an action score , assessing whether the predictions accurately capture the actions described in GT AD, independent of character information. Moreover, as ADs are time-limited, there is always a choice on which visual aspect (characters, actions, objects, etc.) should be included. Consequently, a single video clip can have multiple equally valid ADs, each highlighting a different aspect. This observation is supported by the inter-rater agreement experiments in AutoAD-III [ 28 ] and previous user studies [ 88 ] . Therefore, beyond evaluating single AD performance, we assess our framework’s capability as an assistant to generate multiple AD candidates, and report the performance of the selected one AD. In summary, we propose an enhanced training-free AD generation framework, with the following contributions: • We incorporate shot-based temporal context into AD generation via training-free prompting techniques including shot number referral and dynamic frame sampling. • We develop state-of-the-art methods for thread structure and shot scale predictions, and demonstrate that incorporating predicted film grammar knowledge enhances AD generation. • We improve the current AD evaluation by introducing the character-free action score, and a new assistance-oriented evaluation protocol. • Our approach achieves state-of-the-art performance in training-free AD generation, and furthermore, surpasses fine-tuned models on multiple benchmarks. This is the first time a training-free approach has achieved superior performance to fine-tuned methods. 2 Related work Audio Description generation. Efforts have been made to curate Audio Description (AD) datasets for both movies [ 75 , 28 ] and TV series [ 92 ] , with human annotations sourced from platforms such as AudioVault [ 2 ] . For automatic AD generation, prior works [ 26 , 27 , 28 , 50 , 82 , 22 ] fine-tune pre-trained VLMs on AD

Read the complete text