Directing the Narrative: A Finetuning Method for Controlling Coherence and Style in Story Generation
Jianzhang Zhang, Yijing Tian, Jiwang Qu, Chuang Liu · 2026
Read Directing the Narrative: A Finetuning Method for Controlling Coherence and Style in Story Generation by Jianzhang Zhang, Yijing Tian, Jiwang Qu, Chuang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17295.txt.
Introduction
Zhang and co-authors present a two stage framework for story visualization that keeps characters and style consistent across frames. Story images must track an evolving narrative while holding identity and look steady. Their first tool, Group-Shared Attention, lets attention layers share information across samples without loss. Their second tool uses Direct Preference Optimization to align outputs with human taste and narrative standards. On the ViStoryBench benchmark, they report gains of 10 points in character identity and 18.7 in style consistency.
Catalog record
| CREATOR | Jianzhang Zhang, Yijing Tian, Jiwang Qu, Chuang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37721 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Directing the Narrative: A Finetuning Method for Controlling Coherence and Style in Story Generation Jianzhang Zhang, Yijing Tian, Jiwang Qu, Chuang Liu Abstract Story visualization requires generating sequential imagery that aligns semantically with evolving narratives while maintaining rigorous consistency in character identity and visual style. However, existing methodologies often struggle with subject inconsistency and identity drift, particularly when depicting complex interactions or extended narrative arcs. To address these challenges, we propose a cohesive two-stage framework designed for robust and consistent story generation. First, we introduce Group-Shared Attention (GSA), a mechanism that fosters intrinsic consistency by enabling lossless cross-sample information flow within attention layers. This allows the model to structurally encode identity correspondence across frames without relying on external encoders. Second, we leverage Direct Preference Optimization (DPO) to align generated outputs with human aesthetic and narrative standards. Unlike conventional methods that rely on conflicting auxiliary losses, our approach simultaneously enhances visual fidelity and identity preservation by learning from holistic preference data. Extensive evaluations on the ViStoryBench benchmark demonstrate that our method establishes a new state-of-the-art, significantly outperforming strong baselines with gains of +10.0 in Character Identity (CIDS) and +18.7 in Style Consistency (CSD), all while preserving high-fidelity generation. 1 Introduction Figure 1: Visual showcase of consistent character generation. The leftmost column displays the reference input. Subsequent columns demonstrate our model’s capability to synthesize the same character in various complex scenarios, maintaining strictly consistent identity and style. Transforming textual narratives into coherent visual sequences stands as a pivotal challenge in computer vision and text-to-image synthesis. Story visualization demands that a model function as a consistent visual narrator, synthesizing sequential frames that faithfully align with evolving plot descriptions while rigorously preserving character identity and artistic style. This task presents an inherent trade-off between generative diversity and identity consistency: the model must possess the flexibility to render varied scenes and poses without compromising the structural stability required to prevent the protagonist’s appearance from drifting. While recent diffusion transformers (DiTs) demonstrate exceptional visual fidelity, applying them directly to story visualization remains challenging. Since standard diffusion models are trained on independent samples, they lack an intrinsic mechanism to maintain correspondence across a sequence. Consequently, when generating a multi-frame story, these models frequently suffer from identity drift, in which a character’s facial features or clothing gradually mutate. Furthermore, they often exhibit style inconsistency that disrupts the narrative flow ( Zhou et al. 2024 ) . To mitigate these issues, existing methodologies typically rely on two primary paradigms. The first, optimization-based methods such as LoRA ( Hu et al. 2022 ) and DreamBooth ( Ruiz et al. 2023 ) fine-tune weights for specific characters. While effective, these approaches necessitate computationally intensive per-subject optimization, severely limiting their scalability. The second paradigm, encoder-based methods exemplified by IP-Adapter ( Ye et al. 2023 ) , utilizes external vision encoders (e.g., CLIP) to condense reference images into feature embeddings, which are then injected via cross-attention. However, this reliance on external compression often creates an information bottleneck, resulting in the loss of fine-grained identity details and inducing semantic conflict with textual controls. We argue that consistency should be an intrinsic capability learned through direct feature interaction, rather than relying on compressed external injections. To this end, we propose a novel framework that transforms the pre-trained FLUX.1 1 1 1 https://github.com/black-forest-labs/flux backbone into a consistent story visualizer through a progressive two-stage training strategy. In the first stage, we introduce Group-Shared Attention (GSA). Departing from encoder-based paradigms that depend on compressed global embeddings, GSA facilitates lossless internal information flow. By repurposing the self-attention mechanism to share high-resolution visual tokens across images within the same training batch, GSA enables the model to directly attend to the spatial features of reference views. This design ensures fine-grained structural consistency in identity and attire without compromising the model’s adherence to textual prompts. While GSA establishes structural correspondence, standard reconstruction objectives may still yield anatomical artifacts. Therefore, in the second stage, we employ Direct Preference Optimization (DPO) to align the model with human aesthetic standards. Using an expert-curated dataset, we train the model to discriminate between consistent, high-fidelity generations and those exhibiting subtle distortions. This stage functions as a semantic refinement process, enhancing perceptual quality and reinforcing identity constraints. Figure 1 demonstrates the final coherent narratives achieved by our framework. Our contributions are summarized as follows: 1. We propose Group-Shared Attention (GSA), a novel mechanism that enables structured, lossless information flow across samples. Distinct from encoder-based injection, GSA empowers the model to intrinsically learn robust identity preservation directly from high-resolution internal features. 2. We introduce a holistic two-stage training framework that harmonizes structural consistency with human aesthetic preferences. By combining structural pre-training with DPO-based refinement, our strategy effectively balances the trade-off between rigid identity adherence and high-fidelity semantic generation. 3. We achieve new state-of-the-art performance on the ViStoryBench benchmark ( Zhuang et al. 2025 ) . Extensive experiments demonstrate significant improvements of +10.0 in Character Identity (CIDS) and +18.7 in Style Consistency (CSD) compared to strong baselines, setting a new standard for consistent story visualization. 2 Related Works 2.1 Text-to-Image Generation Text-to-image synthesis has evolved from early Generative Adversarial Networks ( Goodfellow et al. 2020 ) to diffusion-based frameworks. While GANs provided initial solutions, they often faced challenges regarding training stability. Denoising Diffusion Probabilistic Models subsequently offered improved stability and generation fidelity. Modern models have further scaled up image synthesis capabilities. DALL-E 2 ( Ramesh et al. 2022 ) and Imagen ( Saharia et al. 2022 ) demonstrated the effectiveness of leveraging large text encoders. Latent Diffusion Models ( Rombach et al. 2022 ) , such as Stable Diffusion, introduced the efficiency of latent space processing, while subsequent iterations like SDXL ( Podell et al. 2023 ) and Pixart- \alpha ( Chen et al. 2023 ) increased model capacity. Most recently, the field has progressed towards Flow Matching combined with Diffusion Transformer architectures. Models such as FLUX.1 leverage rectified flow matching to learn straight transport paths between noise and data. These flow-based transformers offer improved prompt adherence compared to earlier architectures and serve as the foundational backbone for our investigation. 2.2 Image Customization Image customization aims to synthesize specific subjects based on reference images. Optimization-based methods like Textual Inversion ( Gal et al. 2022 ) and DreamBooth ( Ruiz et al. 2023 ) fine-tune embeddings or model parameters. To reduce computational costs, Parameter-Efficient Fine-Tuning methods like LoRA ( Hu et al. 2022 ) were developed to optimize a subset of parameters. Zero-shot methods such as IP-Adapter ( Ye et al. 2023 ) and InstantID ( Wang et al. 2024 ) utilize encoder-based mechanisms to inject identity features directly. Addressing the complexity of multi-subject generation, UNO ( Wu et al. 2025 ) introduces a Model-Data Co-evolution strategy to construct a sophisticated pipeline for generating aligned data. While these approaches have achieved significant success, there remains a need to investigate methods that directly learn from open-domain natural images without relying on extensive synthetic data construction, offering a complementary perspective to existing pipelines. 2.3 Story Visualization Story visualization requires generating sequences of frames that align with a narrative. The primary task is to maintain contextual consistency across time steps. Early RNN-based methods like StoryGAN ( Li et al. 2019 ) and autoregressive diffusion