Bernini: Latent Semantic Planning for Video Diffusion
Bernini Team, Bytedance · 2026
Read Bernini: Latent Semantic Planning for Video Diffusion by Bernini Team, Bytedance (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22344.txt.
Introduction
The Bernini Team and colleagues unify video generation and editing through a split of labour. A multimodal language model plans meaning, while a diffusion model renders pixels from that plan. The planner predicts a semantic code in embedding space, and the renderer builds pixels from the plan plus text features. Meaning is the interface, so the two parts train mostly apart. A new position embedding handles multiple visual inputs. Bernini leads on a wide range of video generation and editing benchmarks.
Catalog record
| CREATOR | Bernini Team, Bytedance |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104482 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bernini: Latent Semantic Planning for Video Diffusion Bernini Team, Bytedance Abstract Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini , a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM’s pretrained understanding translating into strong generalization on challenging editing tasks. \correspondence \checkdata [Project Page] https://bernini-ai.github.io Figure 1 : Video editing leaderboard. Pairwise human preferences on open-ended video editing (no restriction on instruction type). (A) Leaderboard. BT scores [ 5 ] with 95\% bootstrap CIs; Win % =1/(1+10^{-(s-1000)/400}) vs. avg. opponent; W-L-T raw record. (B) Win-rate matrix. W/(W{+}L) ; Bernini: 480p/24 fps; baselines: 720p/24 fps. Figure 2 : Bernini supports diverse video generation tasks within a unified framework, including text-to-video (T2V), subject-to-video (R2V), video editing (V2V), and reference-guided video editing (RV2V). 1 Introduction Multimodal large language models (MLLMs) [ 69 , 11 , 44 ] and diffusion models [ 60 , 36 , 20 , 53 , 70 ] have matured along largely independent trajectories. Modern MLLMs read long instructions, reason over multiple reference images, and ground their answers in a complex multimodal context. Diffusion models, meanwhile, have become the default tool for photorealistic image and video synthesis at high resolutions and long durations. The natural next step is to combine these two mature families into a single system that both understands intent and generates the desired output, supporting unified understanding, generation, and editing within one model. However, how to do so effectively remains an open question. Our approach begins with two simple observations. First, MLLMs are naturally suited to semantic reasoning : interpreting long instructions, grounding on multiple references, and forming an internal representation of what the output should be. Second, diffusion generation decomposes cleanly into semantic guidance and detail preservation . The high-level content is determined by a compact semantic signal, while fine-grained fidelity, and in editing also consistency with the source input, requires dense pixel-level latents such as VAE features. Crucially, the semantic signal itself need not be high-resolution to be effective. A handful of semantic tokens are enough to specify an entire scene. These observations suggest a clean division of labor: let the MLLM carry out semantic reasoning, and let the diffusion model focus on synthesis, using semantic features as its primary condition and pixel-level features only where detail preservation demands them. A natural question is what representation should carry the semantic signal between the two. We anchor this interface to a representation that already exists within MLLM itself, namely its own ViT embedding space [ 19 , 58 , 71 ] . The MLLM already reasons and represents visual content in this space, so training it to plan the target in ViT embeddings aligns naturally with its pretrained representations and requires minimal adaptation. We instantiate this principle as Bernini , a unified framework for multimodal understanding, generation, and editing. Bernini consists of a planner , based on an MLLM, that predicts the target’s visual representation in the continuous ViT embedding space. Following a masked generative modeling paradigm [ 39 ] , a lightweight ViT embedding decoder on top of the MLLM recovers randomly masked target ViT tokens from the hidden states of the MLLM, and at inference progressively fills in the full target representation from fully masked tokens. The renderer , a Diffusion Transformer (DiT) [ 55 ] , then synthesizes the final image or video through flow-matching [ 43 ] denoising over VAE latent tokens, conditioned on the semantic embedding from the planner through cross-attention and augmented with text features. For editing tasks, VAE features of the source input are additionally injected to preserve detail and consistency. To unify different task types, we adopt a shared input protocol across text-to-video, subject-to-video, and editing, achieving broad modality coverage without task-specific architectures. For multiple visual sources within a unified sequence, we further introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE) , which augments standard spatiotemporal rotary embeddings [ 64 ] with a segment-index-conditioned phase modulation. Finally, to amplify the contribution of understanding to generation, the planner is equipped with a Chain-of-Thought (CoT) mechanism [ 77 , 28 ] that performs reasoning in latent space before producing the final embedding. Because semantics serve as the interface, the two components can be trained largely independently and only lightly co-trained thereafter, preserving the MLLM’s pretrained capabilities and allowing its multimodal understanding to transfer directly into diverse downstream generation tasks. Our contributions are summarized as follows: • We propose Bernini , a unified framework for generation, and editing that uses the MLLM’s own ViT embedding space as a semantic bridge to the diffusion generator, allowing pretrained understanding to transfer directly into generation and enabling strong generalization across diverse video tasks. • We design a suite of data construction pipelines that yield a large-scale, multi-task corpus for unified video generation and editing, spanning video- and image-pair pretraining data, high-quality propagation-based and motion-aware video editing data, and reference-image- and reference-video-guided generation data, providing the diverse and high-fidelity supervision required to train Bernini across all tasks. • Bernini achieves state-of-the-art performance across a wide range of video generation, editing, and subject-to-video benchmarks, including OpenVE-Bench, OpenS2V-Eval, and our newly proposed Bernini-Bench. Figure 3 : Overview of Bernini. (A) Visual and text inputs are serialized into a unified 1D sequence. (B) The MLLM planner predicts target semantic embeddings from masked targets and conditions the renderer. (C) The DiT-based renderer performs flow matching in the VAE latent space conditioned on semantic embeddings and source VAE features. (D) Bernini uses a segment-wise hybrid attention mask in MLLM. (E) Segment-aware 3D RoPE disambiguates visual tokens from different segments. 2 Methods 2.1 Architecture As illustrated in Fig. 3 , Bernini consists of two main components: an MLLM-based planner and a DiT-based renderer. Taking multimodal conditions as input, the MLLM performs multimodal understanding and semantic reasoning to produce the desired target content. An MLP connector then maps these hidden states into the conditioning representation required by the DiT-based renderer. Conditioned on these semantic features, together with additional text features and source visual conditions when available, the DiT-based renderer synthesizes the final image or video in the VAE latent space. 2.1.1 MLLM-based Planner Unified Input Formulation. To support diverse tasks within a single framework, Bernini adopts