Co-Director : Agentic Generative Video Storytelling

Yale Song, Yiwen Song, Nick Losier, Nathan Hodson, Ye Jin, Rhyard Zhu, Yan Xu, Daniel Vlasic, Carina Claassen, Jasmine Leon, Khanh G. LeViet, Zack Chomyn, Joe Timmons, Brett Slatkin, Scott Penberthy, Tomas Pfister · 2026

Read Co-Director : Agentic Generative Video Storytelling by Yale Song, Yiwen Song, Nick Losier, Nathan Hodson, Ye Jin, Rhyard Zhu, Yan Xu, Daniel Vlasic, Carina Claassen, Jasmine Leon, Khanh G. LeViet, Zack Chomyn, Joe Timmons, Brett Slatkin, Scott Penberthy, Tomas Pfister (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24842.txt.

Introduction

Song and colleagues present Co-Director for agentic video storytelling. Current pipelines chain modules with hand-made prompting. They suffer drift and cascading failures. Co-Director treats storytelling as a global optimization problem. A bandit picks good creative directions. A local loop limits identity drift and keeps order. This weighs new narrative plans with good configurations. GenAD-Bench offers 400 fictional product cases for ads. Tests show it beats top rivals and generalizes broadly.

Catalog record

CREATORYale Song, Yiwen Song, Nick Losier, Nathan Hodson, Ye Jin, Rhyard Zhu, Yan Xu, Daniel Vlasic, Carina Claassen, Jasmine Leon, Khanh G. LeViet, Zack Chomyn, Joe Timmons, Brett Slatkin, Scott Penberthy, Tomas Pfister
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105163
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24842

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Co-Director : Agentic Generative Video Storytelling Yale Song, Yiwen Song, Nick Losier, Nathan Hodson, Ye Jin, Rhyard Zhu, Yan Xu, Daniel Vlasic, Carina Claassen, Jasmine Leon, Khanh G. LeViet, Zack Chomyn, Joe Timmons, Brett Slatkin, Scott Penberthy, Tomas Pfister Abstract While diffusion models generate high-fidelity video clips, transforming them into coherent storytelling engines remains challenging. Current agentic pipelines automate this via chained modules but suffer from semantic drift and cascading failures due to independent, handcrafted prompting. We present Co-Director , a hierarchical multi-agent framework formalizing video storytelling as a global optimization problem. To ensure semantic coherence, we introduce hierarchical parameterization: a multi-armed bandit globally identifies promising creative directions, while a local multimodal self-refinement loop mitigates identity drift and ensures sequence-level consistency. This balances the exploration of novel narrative strategies with the exploitation of effective creative configurations. For evaluation, we introduce GenAd-Bench , a 400-scenario dataset of fictional products for personalized advertising. Experiments demonstrate that Co-Director significantly outperforms state-of-the-art baselines, offering a principled approach that seamlessly generalizes to broader cinematic narratives. Project Page: https://co-director-agent.github.io/ 1 Introduction Generative video storytelling aims to transform abstract ideas—such as a simple tagline and sparse reference visuals—into cohesive video narratives, like a fully realized short film. It has diverse applications, ranging from democratizing high-end content creation ( 12 ) to automating filmmaking ( 48 ) and digital advertising ( 6 ) . Realizing this potential requires jointly optimizing multiple objectives: narrative coherence, visual consistency, and mise-en-scène —the visual language dictating how camera angles, color palettes, and metaphors deliver the intended message ( 47 ) . Recent advancements in video diffusion demonstrate remarkable high-fidelity generation. High-profile collaborations, such as Google DeepMind’s work with Darren Aronofsky on Ancestra ( 2 ) and On This Day… 1776 ( 3 ) , showcase the artistic ceiling when backed by a 200-person creative team. While representing the pinnacle of human-AI collaboration, such resources remain out of reach for independent creators. Without them, individuals must manually bridge the gap between LLM scripting and fragmented generative models, correcting for the consistency errors and identity drift inherent in current architectures ( 13 ) . To mitigate this, the community has pivoted toward agentic pipelines where LLMs orchestrate the multi-step generative process ( 30 ; 34 ; 44 ; 42 ; 24 ) . However, these systems remain susceptible to “cascading failures,” where early errors propagate and break long-horizon consistency ( 50 ) . This reflects the classical credit assignment problem ( 32 ) , as terminal failures are difficult to trace back to specific prompts. Furthermore, existing systems rely on static, handcrafted prompt templates lacking a unified creative vision. Without a mechanism to map script-level changes to downstream generation, the resulting visual output often feels disjointed. Crucially, these approaches fail to capture the serendipity essential to creative production. Human directors often explore counter-intuitive ideas to find “creative gold.” Static, chained pipelines are fundamentally ill-equipped for such exploration, remaining anchored to fixed templates unable to adapt to the non-linear nature of creative success. In this work, we present Co-Director , a multi-agent framework for automating video storytelling. To the best of our knowledge, Co-Director is the first system to formalize generative storytelling as a global optimization problem. Unlike prior agentic pipelines that rely on linear, "waterfall" chains prone to semantic drift, we introduce a hierarchical parameterization strategy steered by Multi-Armed Bandit (MAB) ( 8 ) . Rather than relying on fixed prompt templates, Co-Director utilizes MAB to sample abstract creative trajectories and dynamically injects these into the system prompts of sub-agents. This top-down steering ensures that all-agents—from script-writing to video generation—are orthogonally aligned toward a unified creative intent. This formulation formalizes the creative process as a search for the optimal balance between exploration —venturing into novel, unconventional narrative territories—and exploitation —refining historically effective stylistic configurations. This allows Co-Director to move beyond simple self-correction toward a principled exploration of the creative latent space, identifying the configuration that optimally balances visual fidelity with narrative effectiveness ( 14 ) . Furthermore, we argue that evaluating video storytelling requires a shift from open-ended cinematic metrics towards rigorous, constraint-driven frameworks. We use digital advertising as our demonstration scenario because it demands both sophisticated storytelling (much like short cinematic film) and strict adherence to objective requirements: asset preservation, audience alignment, and value proposition communication ( 39 ) . To this end, we introduce GenAd-Bench , a dataset of 400 scenarios across 200 fictional products. By utilizing fictional entities, we respect copyright constraints while preventing models from defaulting to memorized training priors. We provide a multimodal LLM (MLLM)-based evaluation suite covering Visual Asset Fidelity , Demographic Alignment , Visual Quality , and Marketing Appeal —anchored in the AIDA hierarchy ( 36 ; 7 ) . Our primary contributions are summarized as follows: • We propose Co-Director , a multi-agent architecture for video storytelling that achieves global coherence by integrating multi-armed bandit optimization with a local, MLLM feedback-driven refinement loop. • We introduce the hierarchical parameterization of creative intent, enabling the consistent propagation of visual and narrative styles across discrete generative sub-agents. • We present GenAd-Bench to rigorously evaluate generative video storytelling across 400 scenarios, eliminating memorization bias via the use of fictional brand assets. • We demonstrate our approach significantly outperforming existing foundation models and agentic baselines in asset preservation, contextual alignment, and visual quality. • We open-source our implementation of Co-Director and the GenAd-Bench dataset to facilitate future research in generative video storytelling. 2 Related Work Foundation Models for Video Synthesis While large-scale diffusion models ( 19 ; 9 ; 26 ; 40 ) achieve unprecedented cinematic realism, maintaining long-range consistency remains challenging ( 13 ) . Specialized architectures like Phantom ( 31 ) and MoC ( 10 ) introduce advanced techniques to maintain character consistency, but they remain fundamentally standalone generators. Furthermore, these models serve as powerful building blocks rather than complete end-to-end solutions, lacking the high-level planning and reasoning required to create complex narratives in the resulting video output ( 13 ) . Agentic Frameworks for Video Storytelling To bridge the gap between clip generation and narrative production, recent multi-agent systems utilize LLMs to decompose scripts into shot lists and prompts ( 30 ; 44 ) , targeting domains like personalized vlogs ( 23 ) , music videos ( 38 ) , and short films ( 49 ; 45 ; 42 ; 33 ) . However, these “waterfall” pipelines are susceptible to cascading errors. While frameworks like AniMaker ( 34 ) and MAViS ( 41 ) incorporate local consistency checks, they struggle with global credit assignment ( 32 ) across the agentic stack. To resolve this, Co-Director moves beyond the local-refinement paradigm by introducing a Multi-Armed Bandit (MAB) formulation ( 27 ; 8 ) to perform a principled exploration-exploitation of the creative latent space. This global, hierarchical parameterization resolves the “semantic collisions” between sub-agents that plague current pipelines, preventing identity drift and ensuring horizontal cohesiveness. Benchmarks for Video Storytelling As generative capabilities evolve, evaluation methodologies must adapt to measure higher-order reasoning. Generic video benchmarks ( 25 ; 37 ) focus on prompt alignment and broad aesthetics, lacking the visual and narrative qualities demanded by professional workflows. Furthermore, even recent narrative-focused benchmarks, such as MovieBench ( 43 ) and ViStoryBench

Read the complete text