EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues

Rohit Girmaji, Bhav Beri, Ramanathan Subramanian, Vineet Gandhi · 2025

Read EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues by Rohit Girmaji, Bhav Beri, Ramanathan Subramanian, Vineet Gandhi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02172.txt.

Introduction

Rohit Girmaji, Bhav Beri, Ramanathan Subramanian, and Vineet Gandhi wrote this paper in two thousand twenty-five. It presents EditIQ, a full auto editor that cuts a fixed wide-angle video into a film-like watch, as a crew of cameramen might.

From the still feed, the system first spins off many virtual shots, then picks and joins them by an energy rule that weighs film craft: shot choice, cuts, and flow. Two guides feed the picks: a language model reads the talk to find the beats of the scene, and a saliency model marks what eyes would seek. In a viewer study on stage and school footage, EditIQ beat rival auto cuts. The tale stays whole and smooth.

Catalog record

CREATORRohit Girmaji, Bhav Beri, Ramanathan Subramanian, Vineet Gandhi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues Rohit Girmaji, Bhav Beri, Ramanathan Subramanian, Vineet Gandhi Abstract . We present EditIQ , a completely automated framework for cinematically editing scenes captured via a stationary, large field-of-view and high-resolution camera. From the static camera feed, EditIQ initially generates multiple virtual feeds, emulating a team of cameramen. These virtual camera shots termed rushes are subsequently assembled using an automated editing algorithm, whose objective is to present the viewer with the most vivid scene content. To understand key scene elements and guide the editing process, we employ a two-pronged approach: (1) a large language model (LLM)-based dialogue understanding module to analyze conversational flow, coupled with (2) visual saliency prediction to identify meaningful scene elements and camera shots therefrom. We then formulate cinematic video editing as an energy minimization problem over shot selection, where cinematic constraints determine shot choices, transitions, and continuity. EditIQ synthesizes an aesthetically and visually compelling representation of the original narrative while maintaining cinematic coherence and a smooth viewing experience. Efficacy of EditIQ against competing baselines is demonstrated via a psychophysical study involving twenty participants on the BBC Old School dataset plus eleven theatre performance videos. Video samples from EditIQ can be found at https://editiq-ave.github.io/ . Conference: 30th International Conference on Intelligent User Interfaces; March 24–27, 2025; Cagliari, Italy 30th International Conference on Intelligent User Interfaces (IUI ’25), March 24–27, 2025, Cagliari, Italy DOI: 10.1145/3708359.3712113 ISBN: 979-8-4007-1306-4/25/03 CCS: Information systems Multimedia content creation CCS: Mathematics of computing Combinatorial optimization CCS: Computing methodologies Computational photography CCS: Human-centered computing User studies Figure 1. We present EditIQ , an automated video editing pipeline based on dialogue understanding using LLMs and visual understanding via video saliency. First row presents original video frames input to the pipeline, which generates multiple rushes (depicted in the next two rows). The speaker is denoted by a green arrow in the original frame and the transcript below 3rd row. LLMs are employed to analyze the scene’s narrative, guiding shot selections (highlighted in red on the left for each frame in 4th row). Simultaneously, saliency analysis captures prominent visual scene content, giving alternate shot selections (shown in blue on 4th row right for each frame). Combining the language and visual-based scene understanding results generates optical video shots captured in the 5th row. 1. INTRODUCTION Professional video production of an event like a theatre performance or a quiz show usually requires a team of experienced camera operators to film the scene from multiple angles. These multi-camera recordings, termed rushes , are later compiled through careful manual editing to craft a coherent narrative designed to enhance audience engagement and viewing experience. The editing of these performances is typically performed in chronological order, with the process primarily focused on selecting the most appropriate rush at each moment. However, filming in confined spaces, such as live theatre performances, presents unique challenges, including limited vantage points, the impossibility of performing retakes, and the impracticality of maneuvering bulky equipment, making the task both difficult and demanding. Consequently, the need for (i) a skilled camera crew, (ii) multiple cameras and supporting equipment, and (iii) expert editors significantly increases the complexity and cost of the video production process. Consequently, production houses use a wide-field-of-view static camera positioned at a distance suitable to capture the entire stage. This method is common due to its ease of implementation and ability to capture the entire scene. While effective for archival purposes, wide-angle visuals are ineffective in engaging the audience. The distant camera feed conveys the entire scene but fails to capture close-up details like facial expressions and emotions, which are central to cinematic storytelling. As renowned film editor Thelma Schoonmaker once said, “Close-ups reveal the soul of the character, engaging the audience in a profound way.” Prior automated video editing efforts have sought to transform static wide-angle footage into more engaging content using machine learning and optimization techniques. The goal is to reduce costs and complexity of production, while still delivering quality content. To reduce the reliance on multiple cameramen, Gandhi et al. ( Gandhi et al., 2014 ; Gandhi and Ronfard, 2015 ) proposed a framework for automatically generating multiple clips suitable for video editing by simulating pan-tilt-zoom camera movements within the frame of a single static camera. Moorthy et al. ( Moorthy et al., 2020 ) demonstrated that efficient camera selection can be achieved by leveraging eye-gaze data from users. Their work assumes that humans inherently focus on salient scene aspects, and that gaze can serve as a proxy to localize key scene elements. Though their method produces impressive results, its usefulness is restricted due to reliance on gaze data, which may not always be available. To eliminate the need for auxiliary user data to perform video editing, we present EditIQ , a fully automated multi-camera video production pipeline for staged events, using footage from one or more wide-angle, stationary static cameras. We leverage prior efforts ( Gandhi et al., 2014 ) for simulating multiple virtual cameras, and our focus is on automating the camera selection process. Optimal camera selection demands a nuanced understanding of the scene, capturing key elements such as dialogue, the speaker, and the actors’ actions and reactions. Reaction shots, in particular, are vital in editing as they convey the emotional tone of the scene, enhancing the viewer’s engagement with the scene and their perceptual understanding. While the human gaze may naturally track key scene elements, automating this intricate process poses a significant challenge. EditIQ primarily seeks to leverage advancements in large language models (LLMs) for automated video editing. Recent studies have highlighted strong LLM capabilities for understanding key scene elements such as emotions ( Vishnubhotla et al., 2024 ) , entailment ( Zhou et al., 2024 ) , and co-reference resolution ( Sundar et al., 2024 ) . Our work is the first to demonstrate that these models can be effectively utilized to guide camera selection based on narratives and conversations, and determining which scene elements should be visually emphasized. E.g. , in the exemplar scene illustrated in Figure 1 , a camera following the speaker (or an audio source) would focus solely on the lead singer, missing the non-verbal reactions of other musicians as they are introduced. An LLM instead recognizes that when the lead singer says “J T Thomas on the keyboard,” the visual attentional focus should shift to the addressed person. LLMs nevertheless struggle to capture scene actions not explicitly referenced in dialogue, and multimodal LLMs typically perform frame-level processing, making them less effective at understanding temporal actions. To overcome this limitation, we additionally utilize a saliency prediction architecture trained to model human gaze, identifying key areas of importance in a scene. Specifically, we extend a spatio-temporal action localization backbone ( Pan et al., 2021b ) for video saliency prediction ( Wang et al., 2019 ) . Again referring to Figure 1 , the visual saliency network accurately captures key actions, like the keyboard player’s salute or the lead singer’s hand movement, while referring to the bass guitarist. Once the LLM and visual saliency pipelines enable the identification of key scene elements (and corresponding video shots), we formulate camera shot selection as a discrete optimization problem, where

Read the complete text