Object Isolated Attention for Consistent Story Visualization

Xiangyang Luo 1,2 , Junhao Cheng 3 , Yifan Xie 1 , Xin Zhang 1 , Tao Feng 1,2 , Zhou Liu 1 , Fei Ma 1 2 , Fei Yu 1 · 2025

Read Object Isolated Attention for Consistent Story Visualization by Xiangyang Luo 1,2 , Junhao Cheng 3 , Yifan Xie 1 , Xin Zhang 1 , Tao Feng 1,2 , Zhou Liu 1 , Fei Ma 1 2 , Fei Yu 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23353.txt.

Introduction

Xiangyang Luo and co-authors present a method for drawing story picture sequences in which the same characters look consistent from frame to frame. Open-ended story pictures are hard because characters drift in appearance while scenes must stay natural and fit the plot. Many earlier methods lose the character or blur characters into each other.

Their enhanced transformer splits attention into separate channels. One channel sharpens each character's own features and cuts focus on irrelevant areas. The other handles each character apart, so features never fuse. The design leans on knowledge already inside pre-trained image models. The result is a sequence that stays logical and keeps its cast recognizable.

Catalog record

CREATORXiangyang Luo 1,2 , Junhao Cheng 3 , Yifan Xie 1 , Xin Zhang 1 , Tao Feng 1,2 , Zhou Liu 1 , Fei Ma 1 2 , Fei Yu 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23353

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Object Isolated Attention for Consistent Story Visualization Xiangyang Luo 1,2 , Junhao Cheng 3 , Yifan Xie 1 , Xin Zhang 1 , Tao Feng 1,2 , Zhou Liu 1 , Fei Ma 1 2 , Fei Yu 1 Abstract Open-ended story visualization is a challenging task that involves generating coherent image sequences from a given storyline. One of the main difficulties is maintaining character consistency while creating natural and contextually fitting scenes—an area where many existing methods struggle. In this paper, we propose an enhanced Transformer module that uses separate self attention and cross attention mechanisms, leveraging prior knowledge from pre-trained diffusion models to ensure logical scene creation. The isolated self attention mechanism improves character consistency by refining attention maps to reduce focus on irrelevant areas and highlight key features of the same character. Meanwhile, the isolated cross attention mechanism independently processes each character’s features, avoiding feature fusion and further strengthening consistency. Notably, our method is training-free, allowing the continuous generation of new characters and storylines without re-tuning. Both qualitative and quantitative evaluations show that our approach outperforms current methods, demonstrating its effectiveness. I Introduction Story visualization, the task of generating coherent image sequences from a narrative [ 1 ] , has emerged as a rapidly advancing field at the intersection of computer vision and natural language processing. This task holds immense potential for applications in education [ 2 , 3 ] , entertainment [ 4 ] , and beyond, where visual storytelling can greatly enhance narrative engagement and comprehension. However, generating high-quality, consistent visualizations that accurately reflect the storyline remains a significant challenge, especially when managing multiple characters and complex scenes. Traditional story visualization methods have primarily relied on models trained on specific datasets [ 5 , 6 ] , limiting their generalization capabilities and real-world applicability. For instance, approaches like StoryGAN [ 1 ] , which employ generative adversarial networks [ 7 ] , often struggle to maintain character identity and visual coherence throughout a narrative. While recent advancements in diffusion-based models [ 8 , 9 , 10 , 11 ] have improved image quality, they still face difficulties in generating new characters and scenes without extensive re-tuning. With the rise of pre-trained text-to-image models [ 12 , 13 ] , methods are now starting to achieve open-world comic generation, which holds much greater practical value. The current mainstream approaches to maintaining consistency can be broadly categorized into two types: (1) Specifying character positions with bounding boxes and using IP-Adapters [ 14 ] to manipulate cross attention, aiming to preserve character identity. However, this method struggles to produce natural interactions and reasonable layouts [ 15 , 16 , 17 ] . (2) Achieving character consistency through the concatenation of self attention mechanisms, which, despite its potential, has limited success in preserving character identity and often results in feature confusion among multiple characters [ 18 , 19 , 20 ] . Building on the concept of concatenated self attention [ 21 , 22 ] , we propose a novel approach that alleviates feature confusion and enhances character consistency. We observe that different characters frequently reference each other, and images tend to pay insufficient attention to concatenated features. To address this, we introduce an isolated self attention mechanism [ 23 ] that employs masks to prevent mutual attention between characters and enforces focus on the same character across sequences via cross attention information. Additionally, we design an isolated cross attention mechanism which leverages prior knowledge of reasonable layout composition in diffusion models and generates separate prompts for each character, further improving consistency and reducing feature mixing. Fig. 1: Pipeline of our framework. Given a story, we utilize an LLM agent to decompose it into scene prompts and character prompts (a). These prompts are then fed into a pre-trained diffusion model to generate anime-style images (b). We replace the traditional upsampling Transformer block with an extended Transformer block, which introduces an extended branch (c). In this new branch, we design isolated self attention (e) and isolated cross attention (d) mechanisms, which extract cross attention maps from the original branch to enhance character consistency and reduce feature confusion. Our method operates in a training-free manner, allowing for the continuous generation of new characters and storylines without the need for re-tuning. Our contributions can be summerized as follows: • We design an isolated self attention, which refines attention maps to reduce irrelevant focus and enhances attention on the specific reference character. • We propose an isolated cross attention, which separates character features by utilizing the layout generated by diffusion models and individual character prompts. • Extensive qualitative and quantitative experiments demonstrate the effectiveness of our approach in achieving visually consistent and coherent story visualizations. II Related Work II-A Diffusion Models Diffusion models [ 24 ] revolutionized text-to-image generation through iterative denoising processes. Subsequent work [ 25 , 26 , 27 ] refined this approach using latent spaces to reduce computational costs while maintaining quality. Latent Diffusion Models (LDMs) emerged as a widely adopted framework for efficient synthesis across diverse generative tasks. Diffusion Transformers (DiT) [ 28 ] enhanced this paradigm by replacing U-Net with transformer-based designs, improving training efficiency and scalability. Despite varying structures, these models share the fundamental Transformer Block component, incorporating self-attention for image layout and cross-attention for conditional generation from other modalities. Our method modifies this block to achieve consistent story visualization without training, applicable to any existing diffusion framework. II-B Story Visualization Story visualization [ 1 ] aims to generate coherent image sequences that align with multi-sentence paragraphs, enabling dynamic storytelling. Traditional methods [ 5 , 6 ] often rely on training models on specific datasets annotated with scene or character-level details. While these methods achieve consistency within limited domains, their scalability is constrained by data dependence and lack of flexibility. The emergence of text-to-image diffusion models has opened new avenues for tuning-free story generation. These approaches can be broadly categorized into two directions. The first direction integrates pre-trained models [ 15 , 16 ] like IP-Adapter [ 14 ] with bounding box inputs, enabling multi-character control. However, such methods often struggle to model complex character interactions and suffer from visual artifacts, such as the ’copy paste’ effect. The second direction leverages self-attention mechanisms [ 19 , 18 , 17 ] to concatenate features across sequences, which is also applied to wide field such as video and 3D generation [ 29 , 30 , 31 ] . While this approach reduces the need for explicit annotations, it faces challenges such as feature leakage and inconsistent outputs, particularly in modeling long-range dependencies across storylines. Our work adopts the latter approach and addresses these limitations by isolating the attention mechanism to prevent feature leakage and enhance consistency. This design ensures more robust alignment between textual descriptions and generated visual sequences, enabling more natural and coherent story visualization compared to existing methods. III Method The workflow of our method is illustrated in Fig. 1 . Starting with a given storyline, we utilize a Large Language Model (LLM) [ 32 ] to generate scene prompts S=\{S_{1},S_{2},\dots,S_{N}\} and character prompts C=\{C_{1},C_{2},\dots,C_{K}\} . For each scene S_{i} , we define three sets of indices: \mathcal{I}_{i} , \mathcal{I}_{i}^{\text{new}} , and \mathcal{I}_{i}^{\text{old}} , representing the indices

Read the complete text