A u r o r a : Unified Video Editing with a Tool-Using Agent
Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo · 2026
Read A u r o r a : Unified Video Editing with a Tool-Using Agent by Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18748.txt.
Introduction
Yongsheng Yu and six colleagues build Aurora, a 2026 system that turns rough video edit requests into finished edits. Current unified editors expect ready-made text, reference pictures, and exact locations for local changes, which users rarely supply. A vision-language agent first turns a raw request into a clear edit plan and picks reference pictures. That plan then guides a video diffusion model through replacement, removal, style change, and insertion.
Catalog record
| CREATOR | Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74291 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A u r o r a : Unified Video Editing with a Tool-Using Agent Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo Abstract Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer’s conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. 1]UofR 2]MIT-IBM 3]NVIDIA \metadata [Code] https://github.com/yeates/Aurora \metadata [Homepage] https://yeates.github.io/Aurora-Page Figure 1 : Aurora converts underspecified instructions into model-ready video-editing conditions. The VLM agent invokes web image search to recover the missing reference images for a Heineken neon sign, a Burberry check scarf, and a print of The Great Wave off Kanagawa, and invokes segmentation to recover the missing mask for a person removal. 1 Introduction Recent video editing models increasingly accept several conditioning inputs in one model: text specifies the edit, the source video supplies content to preserve, and optional reference images or masks specify appearance and location [ 36 , 16 , 19 , 39 ] . This design lets one set of weights support replacement, removal, style transfer, and reference-driven insertion, but it also assumes that each edit request is provided with the appropriate conditioning inputs, including a precise text instruction, a reference image that specifies the visual characteristics of a requested real-world entity or visual pattern, and a mask when the edit is spatially localized. However, in practice, user requests are often incomplete with respect to these conditioning channels. We categorize this incompleteness along two axes. Visual underspecification occurs when the instruction specifies an edit target or region without providing the visual input needed to ground it. For example, “Replace the striped shirt held by the woman with a Burberry check scarf” names a specific visual pattern but provides no reference image. Textual underspecification occurs when the desired edit is described only indirectly, as in “Change the blue item to a vehicle that runs much faster” , where the model must infer the concrete editing goal. Thus, unified conditioning video editing models provide channels for text, reference images, and masks, but they do not completely interpret how missing conditions should be inferred, grounded, or filled from underspecified user requests. Figure 1 shows underspecified instructions that need additional conditions. This gap is best understood as a condition-construction problem. Unified video editing models already define the conditioning tuple they expect, but what remains unresolved is how to construct that tuple from a user’s raw request. The missing piece may be a more complete instruction, a reference image that specifies the visual characteristics of a requested real-world entity or visual pattern, or spatial grounding for a local edit. Each arises before video generation begins. Accordingly, we decouple condition construction from video generation. Figure 2 : Aurora pipeline. The Aurora VLM agent (a LoRA-finetuned Qwen3-VL-8B-Instruct [ 24 ] ) parses the raw user request, triggers tools, rewrites the instruction, and routes the task. The output is the unified conditioning tuple (y^{\prime},V_{src},R^{+}) that the Aurora video diffusion transformer consumes; the task label is part of the agent plan and supports the instruction rewrite, but is not fed to the video model. We present Aurora, an agentic video editing framework built around this split between the VLM agent and video model, shown in Figure 2 . Our design has two parts. First, a tool-augmented VLM agent maps a source video and a raw user request to a four-field edit plan: a rewritten instruction, a task label, an optional image-search query, and an optional mask phrase. The first two fields resolve textual underspecification through rewriting and task routing; the latter two recover missing visual conditions through web image search and grounded segmentation. When the source clip and the user’s text already specify the edit, the tool fields remain empty and the framework reduces to ordinary instruction-conditioned editing. Second, Aurora uses a unified video diffusion transformer (DiT) whose conditioning paths match those plan fields. Built on Wan2.2-TI2V-5B [ 34 ] , the Aurora video DiT consumes the resulting conditions along two paths: a shared multimodal context for the rewritten instruction, sampled source frames, and reference images, and a latent token sequence where source and reference tokens accompany the noisy video tokens. The video DiT itself does not rewrite instructions or call tools. It consumes the conditioning tuple produced by the VLM agent. We also introduce AgentEdit-Bench, a video editing benchmark designed to evaluate this two-part framework jointly rather than each part in isolation. Each case is built around a request that is underspecified along either the textual or the visual axis, so the framework succeeds only if it fills in the missing visual input or instruction text with a concrete editing goal. This structure supports three evaluation protocols on the same source clips: our VLM agent with our video DiT, our VLM agent with any unified video editing model, and any video editing model under the raw user instruction alone. The first measures the full framework, the second isolates VLM agent transfer across video editing models, and the third measures how much the VLM agent helps over an instruction-only baseline. Our contributions are as follows. • We present Aurora, an agentic video editing framework that pairs a tool-augmented VLM agent with a unified video diffusion transformer, so that raw user requests are mapped to more complete and model-ready conditions before generation. • We construct VLM agent training data for complete edit planning, reference-image selection, and DPO-based preference alignment. • We introduce AgentEdit-Bench, a benchmark designed to evaluate video editing enhanced by a tool-augmented VLM agent under textual and visual underspecification. 2 Related Work 2.1 Instruction-conditioned and unified video editing Early instruction-conditioned video editing methods adapted image-editing techniques to video by sharing attention across frames [ 23 , 41 , 9 ] . Senorita-2M [ 48 ] scales supervision with task-specialist synthesizers, and OpenVE [ 11 ] and Ditto [ 1 ] place VLMs in the data-construction pipeline to generate instructions and filter low-quality edits. Recent methods move toward unified video editing models that accept richer conditioning inputs, including instructions, source videos, reference images, masks, in-context examples, or reasoning frames [ 15 , 19 , 36 , 16 , 39 ] . As conditioning becomes richer, the bottleneck shifts from what the model can ingest to what the user can reliably provide. Their conditioning interfaces accept references and