StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization
Gopalji Gaur, Mohammadreza Zolfaghari, Thomas Brox · 2025
Read StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization by Gopalji Gaur, Mohammadreza Zolfaghari, Thomas Brox (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03735.txt.
Introduction
Gopalji Gaur, Mohammadreza Zolfaghari, and Thomas Brox wrote this paper in two thousand twenty-five. It keeps the same subject looking the same across a run of story images, with no retraining of the image model.
Text-to-image tools drift: a hero's face or coat shifts from scene to scene. StorySync shares attention across the batch of images, masked so the subject's traits align as they form. A second step, regional feature harmonization, evens out the fine detail that still clashes. The whole trick rides on a frozen pretrained model, so its flair stays intact. Tests across varied tales show steady subjects from frame to frame. The cost is a small fraction of fine-tuning routes.
Catalog record
| CREATOR | Gopalji Gaur, Mohammadreza Zolfaghari, Thomas Brox |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03735
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StorySync: Training-Free Subject Consistency in Text-to-Image Generation via Region Harmonization Gopalji Gaur, Mohammadreza Zolfaghari, Thomas Brox Abstract Generating a coherent sequence of images that tells a visual story, using text-to-image diffusion models, often faces the critical challenge of maintaining subject consistency across all story scenes. Existing approaches, which typically rely on fine-tuning or retraining models, are computationally expensive, time-consuming, and often interfere with the model’s pre-existing capabilities. In this paper, we follow a training-free approach and propose an efficient consistent-subject-generation method. This approach works seamlessly with pre-trained diffusion models by introducing masked cross-image attention sharing to dynamically align subject features across a batch of images, and Regional Feature Harmonization to refine visually similar details for improved subject consistency. Experimental results demonstrate that our approach successfully generates visually consistent subjects across a variety of scenarios while maintaining the creative abilities of the diffusion model. Figure 1: Demonstrating StorySync ’s consistency capabilities. (a) Subject consistency maintained across various subject categories including humans, animals, and fictional characters. (b) Visual story generation showing StorySync’s ability to maintain subject identity throughout multi-scene visual story sequences. 1 Introduction Current text-to-image diffusion models [ 33 , 2 , 36 ] struggle with maintaining subject consistency when generating multiple images. The lack of subject consistency in visual story generation extends beyond storytelling applications. In fields such as animation, game design, video creation, and synthetic data creation, consistent character representations are crucial for achieving coherence and realism. Various recent research efforts have explored methods to address this problem. Most methods follow the idea of checkpoint personalization [ 38 , 6 , 22 , 3 , 24 ] , where the model is finetuned to generate a consistent subject. However, these approaches typically require extensive subject-specific training and struggle with incorporating multiple subjects in the same image [ 38 ] . Other finetuning-based story generation approaches finetune either components of the diffusion model or an external layer to learn to generate consistent subjects [ 46 , 16 , 55 ] . However, all these approaches require additional training steps and suitable training data. In contrast, training-free methods, such as the IP-Adapter [ 50 ] use an image-conditioned diffusion process that takes a reference image as input and generates similar output images. Encoder-based approaches [ 1 , 12 , 23 , 7 ] focus on aligning the output image with a reference target, which restricts the model’s creative potential and produces images that do not closely follow the input prompts, therefore limiting creativity and preventing characters from adapting dynamically to new scenes. Existing training-free consistent subject generation approaches [ 43 , 54 ] share visual knowledge about the subject among all images through attention sharing. Instead of relying on personalization or reference-based alignment, they leverage cross-image feature sharing to enforce zero-shot subject consistency in a batch of generated images. Other methods implement additional modifications such as text embedding weighting [ 27 ] , or embedding clustering to generate visually similar subjects [ 3 ] . Despite achieving impressive subject consistency, these methods either fail to adhere to the conditioning prompts, or they lack alignment of finer details of the consistent subjects. In this work, we introduce StorySync , which is built on three technical innovations . (1) We introduce masked cross-image attention sharing, a dense interaction of attention features localized to subject regions in the images. (2) To improve on the consistency of subtle subject details, we introduce Regional Feature Harmonization. (3) We present Base Layout Interpolation to enable sufficient diversity in the generated images, despite the consistency constraints. As a result, StorySync is able to generate story scenes with a high level of visual consistency of the story characters and surpasses existing training-free approaches, as shown in Figure 1 . Furthermore, we demonstrate the plug-and-play capability of this approach on different text-to-image diffusion models. We test the approach with SDXL [ 33 ] and Kandinsky 3 [ 2 ] , both of which build on U-Net [ 37 ] , as well as with the time-distilled FLUX.1-schnell model based on a transformer model. 2 Related Work Encoder-based approaches There have been multiple approaches [ 20 , 40 , 30 , 53 , 13 ] to solve the challenge of generating visual stories using Diffusion Models [ 17 ] . These approaches collectively address critical challenges such as maintaining coherence across scenes and ensuring character consistency in the generated scenes. Ye et al. [ 50 ] introduced IP-Adapter , a text-compatible image prompt adapter to enhance alignment between text and visuals, while Wei et al. [ 47 ] proposed ELITE , which encodes visual concepts into textual embeddings for customized T2I generation. Jeong et al . [ 20 ] introduced a zero-shot framework leveraging Latent Diffusion Models and textual inversion to generate coherent storybooks directly from textual inputs. Building on this work, other approaches introduced frameworks for disentanglement of character and scene generation [ 40 ] , or using a history-aware auto-regressive latent diffusion model [ 30 ] to produce cohesive visual narratives. These contributions, although successful in generating cohesive and consistent storyboards, depend largely on input reference images to influence the storyboard generation. Model Finetuning Subsequent works have focused on model personalization to generate consistent subjects in diffusion models [ 56 , 6 , 12 , 19 , 52 , 10 , 35 ] . For example, Richardson et al. [ 35 ] embedded novel concepts into a model’s knowledge space, enabling consistent concept generation, while Arar et al. [ 1 ] fine-tuned models with real-world concepts in just 12 steps. Key approaches like Textual Inversion [ 11 ] , Dreambooth [ 38 ] , and Custom Diffusion [ 22 ] associate new concepts with unique tokens in the text encoder dictionary, allowing models to reconstruct these concepts during generation. Gong et al. [ 13 ] developed TaleCrafter for interactive visual storytelling using customized LoRA [ 18 ] models, while Yang et al. [ 49 ] trained multi-modal LoRA models for consistent subject generation in long stories. Fine-tuning T2I models on storyboard datasets has also shown promise for visual storytelling [ 8 , 25 ] . Wu et al. [ 48 ] extended this approach to video generation, achieving temporal consistency with personalized T2I models. Decentralized methods [ 14 , 32 ] utilize ensemble models to generate multi-subject consistent scenes. Wang et al. [ 44 ] further advanced this by sampling latent noise from localized regions of the latent space, enabling consistent character generation. However, these approaches require computationally expensive fine-tuning of model checkpoints, limiting practical applications. In addition to addressing coherence, multi-modal frameworks such as SEED-Story by Yang et al. [ 49 ] , TaleCrafter by Gong et al. [ 13 ] and Liu et al.’s Intelligent Grimm [ 24 ] push the boundaries of interactive and multi-modal storytelling using Latent Diffusion Models [ 36 ] . Together, these works offer robust solutions for generating visually similar subjects, but often require fine-tuning the diffusion models to generate visually consistent subjects. Figure 2: Proposed Architecture: (1) We cache queries generated by the base model during image generation, (2) We modify the image generation pipeline’s attention processors with our own implementations, (3) We generate images with consistent subjects using the modified