PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with a New Benchmark Dataset

Samuel Teodoro, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim, School of Electrical Engineering, Korea Advanced Institute of Science, Technology · 2024

Read PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with a New Benchmark Dataset by Samuel Teodoro, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim, School of Electrical Engineering, Korea Advanced Institute of Science, Technology (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12877.txt.

Introduction

Samuel Teodoro and co-authors present a film editing tool in a 2024 paper that changes named items and leaves the rest alone. Text-guided editing has made film fixes fast, but most tools shift the whole frame or just one item. Edits can leak into spots the user meant to keep.

Their method shifts chance weight inside the frame so each named item gets its own clean edit zone. It needs no new training on the film at hand. The team also built a fresh test set and marks to grade such edits, since old ones fit poorly. Editors gain a way to fix two or more items in one clip with less spill.

Catalog record

CREATORSamuel Teodoro, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim, School of Electrical Engineering, Korea Advanced Institute of Science, Technology
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-12877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with a New Benchmark Dataset Samuel Teodoro, Agus Gunawan, Soo Ye Kim, Jihyong Oh, Munchurl Kim, School of Electrical Engineering, Korea Advanced Institute of Science, Technology Abstract Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the editing process. However, recent zero-shot video editing techniques focus on global or single-object edits, which can lead to unintended changes in other parts of the video. When editing multiple objects only within their localized regions, existing methods face challenges, such as unfaithful editing, editing leakage, and lack of suitable evaluation datasets and metrics. To overcome these limitations, we propose P robability R edistribution for I nstance-aware M ulti-object Video Edit ing ( PRIMEdit ). PRIMEdit is a zero-shot framework that introduces two key modules: (i) Instance-centric Probability Redistribution (IPR) to ensure precise localization and faithful editing and (ii) Disentangled Multi-instance Sampling (DMS) to prevent editing leakage. Additionally, we present the MIVE dataset for multi-instance video editing, including its subsets categorized by instance counts and sizes for controlled analysis across different levels of scene complexity. We also introduce the Cross-Instance Accuracy (CIA) Score that is firstly proposed to evaluate the editing leakage in multi-instance video editing tasks, which exhibits strong correlation with human evaluation as measured by Spearman’s ρ = 0.86 \rho=0.86 . Our extensive qualitative, quantitative, and user study evaluations demonstrate that PRIMEdit significantly outperforms recent state-of-the-art methods in terms of editing faithfulness, accuracy, and leakage prevention, setting a new benchmark for multi-instance video editing. PubID: pubid: © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Published in IEEE Trans. Pattern Anal. Mach. Intell., vol. 48, no. 11, pp. 15571–15588, Nov. 2026. DOI: 10.1109/TPAMI.2026.3722239 I Introduction The popularity of short-form social media videos has grown significantly [ 1 , 2 ] , yet editing them remains time-consuming [ 3 ] and often requires professional assistance [ 4 ] . These challenges have driven the development of AI-based video editing (VE) tools [ 2 ] , enabled by advances in generative [ 5 , 6 , 7 ] and visual language models [ 8 ] . By allowing users to specify edits through simple text prompts [ 9 , 10 ] , these tools make editing faster and accessible. † † *Co-first authors (equal contribution) † † †Co-corresponding authors † † We provide sample input and edited videos for our method along with baseline methods in our project page: https://kaist-viclab.github.io/primedit-site/ Recent VE methods leverage pre-trained text-to-image (T2I) models [ 5 ] . Compared with training on large-scale datasets [ 11 , 12 ] or fine-tuning on single videos [ 13 , 14 , 9 ] , zero-shot methods [ 15 , 16 , 17 , 18 , 19 , 10 , 20 , 21 ] continue to gain attraction for their efficiency and the availability of pre-trained T2I models. Most zero-shot approaches perform global editing that modifies the entire scene [ 15 , 18 ] or single-object editing that may unintentionally affect other parts of the video [ 22 , 14 ] . However, users often require precise object-level editing without altering the rest of the video, e.g., replacing explicit content, like cigarette, to create family-friendly versions [ 23 ] . Local VE addresses this problem by accurately manipulating specific objects in videos. However, adopting pre-trained T2I models for this task is challenging, as they lack fine-grained control and require additional training [ 11 ] or attention manipulation [ 24 , 14 ] for spatial control. The problem is exacerbated when models perform multiple local edits simultaneously using a single long caption. Fig. 1 illustrates the limitations of previous SOTA VE methods. As shown in Fig. 1 -(a), ControlVideo [ 15 ] , due to the lack of explicit local control, suffers from (i) unfaithful editing (e.g., the red bag not transforming into a yellow purse) and (ii) attention leakage [ 25 ] , where edits intended for one object unintentionally affect others (e.g., both wall and statue becoming yellow). Ground-A-Video (GAV) [ 20 ] in Fig. 1 -(b) shows that simultaneous multi-object VE is feasible using a T2I model [ 5 ] fine-tuned with grounding conditions [ 26 ] . However, GAV also suffers from attention leakage [ 25 ] , especially when object bounding boxes overlap (e.g., the statue also gaining a flower crown). Very recently, VideoGrain [ 24 ] attempted to address attention leakage using object masks and attention modulation [ 27 ] . However, modulation in spatio-temporal attention (STA) introduces significant memory and computational overhead, especially as the number of video frames increases (see Table XII ). Moreover, to describe edited scenes, VideoGrain relies on global edit captions, which may be unavailable or difficult to generate for scenes with a large number of objects. Furthermore, both GAV [ 20 ] and VideoGrain [ 24 ] are evaluated on limited datasets, insufficient for comprehensive testing of multi-instance VE across diverse viewpoints, instance sizes, and numbers of instances. In addition, the global editing metrics used in these methods fail to accurately measure local editing quality, which is essential for multi-instance VE. Fig. 1: Limitations of previous SOTA methods. (a) ControlVideo [ 15 ] relies on single global captions, and (b) GAV [ 20 ] depends on bounding box conditions that can sometimes overlap. Both are susceptible to unfaithful editing ( red arrow ) and attention leakage ( blue arrow ). Note that the colored words in Prompt correspond to the instances to be edited in Input . From the above, four critical challenges remain in multi-instance VE: (i) Attention leakage . The lack of local control in pre-trained T2I models, imprecise input conditions [ 20 ] , and reliance on a single global edit caption [ 15 ] prevent previous methods from effectively disentangling edit prompts; (ii) Unfaithful editing . The absence of techniques to enhance editing faithfulness in multi-instance VE often leads to inaccurate edits; (iii) Lack of fast and intuitive multi-instance VE methods . Although modulation in [ 24 ] enables per-object editing, it slows the generative process, and reliance on full-scene descriptions makes editing cumbersome, especially in cluttered videos. (iv) Lack of evaluation datasets and metrics . Recent methods [ 24 , 20 ] are evaluated on limited datasets using metrics inadequate for assessing local VE quality. To address these challenges, we present P robability R edistribution for I nstance-aware M ulti-object Video Edit ing ( PRIMEdit ), an intuitive zero-shot multi-instance VE framework that achieves faithful edits and reduces attention leakage by disentangling multi-instance edits. PRIMEdit readily adopts existing T2I and T2V models within a unified framework and achieves multi-instance editing through two key modules: (i) To enhance editing faithfulness and increase the likelihood of objects appearing within their masks, we introduce Instance-centric Probability Redistribution (IPR, Sec. III-C ) in the cross-attention layers; and (ii) Inspired by prior works [ 28

Read the complete text