Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation

M. Burak Kizil, Enes Sanli, Niloy J. Mitra, Xuelin Chen, Erkut Erdem, Aykut Erdem, Duygu Ceylan · 2026

Read Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation by M. Burak Kizil, Enes Sanli, Niloy J. Mitra, Xuelin Chen, Erkut Erdem, Aykut Erdem, Duygu Ceylan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01900.txt.

Introduction

Kizil and co-authors give video models a filmmaker's sense of framing. Generative video now looks sharp and coherent in time. Yet deliberate camera control stays out of reach. Camera motion is often a random byproduct of synthesis. It ignores the human subject who drives the scene.

Their Auteur method frames shots relative to the actor. Shot size, angle, and composition follow pose and motion. A human-centric parameterization captures that idea formally. A domain-specific language encodes the framing choices. It converts to standard six-degree camera parameters. A tuned multimodal model acts as virtual director. It maps text and rough motion to sparse keyframes.

Catalog record

CREATORM. Burak Kizil, Enes Sanli, Niloy J. Mitra, Xuelin Chen, Erkut Erdem, Aykut Erdem, Duygu Ceylan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01900

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation M. Burak Kizil, Enes Sanli, Niloy J. Mitra, Xuelin Chen, Erkut Erdem, Aykut Erdem, Duygu Ceylan Abstract Generative video models have achieved remarkable visual fidelity and temporal coherence, yet intentional camera control remains elusive. Existing frameworks treat camera motion as a byproduct of pixel synthesis, producing trajectories that are stochastic, spatially inconsistent, and indifferent to the human subject driving the scene. In this work, we present Auteur, a method for language-driven, human-centric camera framing in generative video. Our core insight is that professional filmmakers conceive shots not as world-space trajectories but as framings defined relative to the actor, encoding shot size, angle, and composition as functions of human pose and motion. We formalize this intuition as a human-centric camera parameterization and introduce a Domain-Specific Language (DSL) that is convertible to standard 6-DoF camera parameters. A fine-tuned multimodal large language model then acts as a virtual director , mapping natural language descriptions and coarse human motion to sparse DSL keyframes that are deterministically interpolated into continuous camera trajectories, which are then provided as input to video generators. We train and evaluate Auteur on a new dataset of 34K aligned text, human motion, and DSL-annotated camera trajectories drawn from procedural synthesis and real-world movie footage from the CondensedMovies ( 3 ) dataset. Auteur enables cinematographic framing of human-centered scenes, a capability largely absent in prior generative models. To assess this behavior, we propose new framing-focused metrics, and our experiments show that Auteur consistently outperforms existing methods. “There are no good and bad movies, only good and bad directors.” — François Truffaut 1 Introduction The essence of cinematography lies not in simply recording a scene but in intentionally orchestrating perspective to evoke narrative meaning. While recent generative video models ( 24 ; 33 ; 40 ; 16 ) have made remarkable strides in visual fidelity and temporal coherence, they lack the structured, deliberate camera behavior required to tell a coherent visual story. Current methods for camera control in video generation enable direct conditioning on explicit 6-DoF geometric trajectories ( 17 ; 8 ) but specify them in world-space coordinates, decoupled from the actors on-screen. Often, transferring a trajectory extracted from reference footage to a new scene produces shots that are geometrically plausible but narratively meaningless; the camera moves, but not in response to the subject’s performance. Object-centric approaches ( 23 ) and sampling-based alternatives ( 10 ) reduce subject-frame violations but still do not capture the deliberate synchronization between actor dynamics and camera response that defines cinematography. Directors of photography do not conceive of shots as mere raw spatial trajectories ( 31 ; 22 ; 4 ) . Instead, they reason in terms of framing : shot size (close-up, medium, wide), camera angle (eye-level, low, overhead), and screen composition (lead room, headroom), all defined relative to the actor’s body and movement over time. Cinematography is therefore not an independent geometric construct, but a function of the subject’s position, movement, and narrative role within the scene. We argue that a generative system capable of authentic cinematography must transition from a passive observer to an active auteur : an agent that deliberatively authors camera behavior with awareness of the humans and their space-time choreography. Figure 1: Auteur defines every camera decision relative to the human subject: how much of the body is visible ( scale ), from which angle the actor is viewed ( orientation ), and how they are placed in the frame ( composition ). Given a natural-language description (top), a fine-tuned multimodal LLM generates a structured DSL program (both actor and camera DSLs) that encodes these actor-relative parameters at sparse keyframes, which are then deterministically interpolated into a continuous 6-DoF camera trajectory conditioned on the human mesh (blue). The trajectory then guides video synthesis. Building on this insight, we introduce Auteur, a method for composing human-aware camera motion from natural language. Our approach is grounded in three key components: a human-centric camera representation, a two-stage generative formulation, and a dataset that anchors both in real or synthetic footage. At the core of Auteur is a human-centric camera parameterization that represents camera state along semantic axes, namely orientation, shot scale, camera level, framing, and look-at level, all defined relative to the actor’s body coordinate frame derived from SOMA ( 38 ) . We implement this continuous parameterization as a Domain-Specific Language (DSL): a structured, discrete form that is human-readable and interpretable, generated by a large language model, and can be deterministically converted to standard 6-DoF camera parameters. The DSL can be viewed as a quantization of the underlying parameterization, mapping a continuous space into a vocabulary of cinematographically meaningful values while preserving full geometric expressiveness. Building upon this representation, Auteur operates in two stages. First, a fine-tuned multimodal LLM, Qwen-2.5-VL ( 35 ) maps a natural language description to a coarse 3D human trajectory, capturing the subject’s spatial evolution. Second, conditioned on this trajectory, the LLM acts as a virtual director, generating DSL programs that specify camera state at sparse, scene-defining keyframes, such as shot initiations and transitions. These keyframes are deterministically interpolated into a continuous camera trajectory that is both geometrically consistent and cinematographically meaningful, and is used to condition downstream video generation models. To train Auteur, we construct a dataset combining sythetic procedural and real-world footage. For procedural data, we synthesize scenes by pairing randomized SOMA ( 38 ) motion sequences with DSL-driven camera programs, rendering them via a 3D engine to obtain (human motion, camera trajectory, DSL program) triplets. For real-world footage, we process videos from CondensedMovies ( 3 ) to recover 4D human motion and camera trajectories ( 42 ) ; captions are generated with AuroraCap ( 6 ) , and DSL annotations are obtained via our motion-tagging strategy. This yields 34K aligned (caption, 3D human motion, camera trajectory, DSL program) tuples that ground our learning of human-aware cinematic priors in real filmmaking practice. We supply the generated camera trajectories by Auteur to multiple downstream video generation models, demonstrating improved spatial coherence, subject alignment, and perceived cinematic quality over prior methods. In summary, our contributions are: • Human-centric camera parameterization and DSL. An actor-relative representation of camera state grounded in professional cinematographic conventions, with framing , rather than raw trajectory, as the primary compositional primitive. The DSL is its discrete LLM-generatable operationalization, deterministically convertible to standard 6-DoF parameters. • Language-to-human-to-camera pipeline. A two-stage method mapping natural language to coarse 3D human trajectories, then to sparse DSL keyframe programs that are deterministically interpolated into continuous, actor-aware 6-DoF camera paths. • Auteur dataset. A dataset of 34K (caption, SOMA parameters over time, camera trajectory, DSL program) samples from real data, enabling learning of human-aware cinematic priors. • Comprehensive cross-dataset evaluation We introduce Auteur Score, a new framing-centric benchmark suite measuring subject visibility, compositional adherence, temporal framing stability, and actor-camera coordination. Across both our internal Auteur benchmark and the external PulpMotion benchmark, our method outperforms prior controllable camera baselines ( 23 ; 10 ) , achieving stronger controllability, more stable cinematic framing 2 Related Work Explicit camera conditioning in generative video. Large-scale video diffusion models (e.g., Sora, Wan, Veo) produce compelling photorealistic visual priors of scene content but treat viewpoint as an uncoupled, passive

Read the complete text