Sissi: Zero-shot S tyle-guided I mage S ynthesis via S emantic–style I ntegration
Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin · 2026
Read Sissi: Zero-shot S tyle-guided I mage S ynthesis via S emantic–style I ntegration by Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06605.txt.
Introduction
Yingying Deng and 4 co-authors present a 2026 arXiv paper on giving generated images a chosen visual style. Their system, called Sissi, takes a reference style image and a text prompt. It then makes a new picture that follows the prompt while copying the reference style. The work targets a common weakness of diffusion models, which are systems that build images from noise guided by text.
Sissi needs no extra training for each new style. It places the style image beside a partly hidden target image and lets a pretrained filling-in model merge them. A new mechanism adjusts how strongly the model listens to text versus style, which reduces conflicts between the two. Tests report faithful styling with better balance and cleaner results than older methods.
Catalog record
| CREATOR | Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06605
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sissi: Zero-shot S tyle-guided I mage S ynthesis via S emantic–style I ntegration Yingying Deng, Xiangyu He, Fan Tang, Weiming Dong, Xucheng Yin Abstract Text-guided image generation has advanced rapidly with large-scale diffusion models, yet achieving precise stylization with visual exemplars remains difficult. Existing approaches often depend on task-specific retraining or expensive inversion procedures, which can compromise content integrity, reduce style fidelity, and lead to an unsatisfactory trade-off between semantic prompt adherence and style alignment. In this work, we introduce a training-free framework that reformulates style-guided synthesis as an in-context learning task. Guided by textual semantic prompts, our method concatenates a reference style image with a masked target image, leveraging a pretrained ReFlow-based inpainting model to seamlessly integrate semantic content with the desired style through multimodal attention fusion. We further analyze the imbalance and noise sensitivity inherent in multimodal attention fusion and propose a Dynamic Semantic-Style Integration (DSSI) mechanism that reweights attention between textual semantic and style visual tokens, effectively resolving guidance conflicts and enhancing output coherence. Experiments show that our approach achieves high-fidelity stylization with superior semantic-style balance and visual quality, offering a simple yet powerful alternative to complex, artifact-prone prior methods. keywords Zero-shot, Style-guided, Image synthesis, Style transfer 1 Introduction Figure 1 : Different pipelines for style-guided image synthesis: (a) style-related text-guided image synthesis; (b) training-based style-guided image synthesis (InstaStyle 7 ); (c) training-free style-guided image synthesis (StyleAligned 19 ); and (d) our method, which simply leverages in-context learning in a pre-trained ReFlow inpainting model enhanced by dynamic semantic-style integration for robust fusion. Figure 2 : Examples of style-guided image synthesis using our method Sissi . Given a style reference, our approach generates stylized images conditioned on a content prompt, achieving exceptional quality and fidelity. Furthermore, it seamlessly extends to multiple style-guided synthesis and image editing tasks, demonstrating versatility and superior performance. Text-guided image generation has gained increasing attention due to the rise of large-scale diffusion models, which allow users to specify desired styles through textual prompts. However, relying solely on text for style guidance presents several limitations. As shown in Fig. 1 (a), the abstractness and ambiguity of textual style descriptions make it difficult to capture fine-grained visual details such as brush strokes, textures, subtle color variations, material gloss, or specific artistic effects (e.g., impasto in oil paintings). Moreover, a short prompt often cannot fully convey these details, potentially resulting in style transfer outputs that deviate from user expectations. To overcome these limitations, recent research has incorporated style reference images to provide explicit visual examples of target styles. A set of methods 54 ; 7 directly learn the corresponding textual embedding from images, using the expressive power of the textual space to convey the stylistic characteristics of artistic images. Adapter-based fine-tuning for text-to-image diffusion models 32 ; 4 integrate style control through a learned adapter module, and encode the style image to style textual embeddings which is also concatenated to text embedding to guide the stylized results. However, the image-guided approaches mentioned above typically require task-specific training and exhibit limited generalization to unseen styles. Moreover, their tendency to entangle structural details from the reference style image often leads to undesired content distortions and overfitting (see Fig. 1 (b)). Alternatively, training-free inversion-based methods 6 ; 16 ; 19 leverage DDIM inversion 44 and latent feature fusion to inject visual style from a reference image into the text-conditioned latent space. While avoiding retraining, these approaches are computationally expensive during inference and sensitive to inaccurate attention estimation, often producing artifacts or unstable style alignment (see Fig. 1 (c)). Collectively, existing approaches struggle to simultaneously achieve high visual quality, strong style–content consistency and robustness. Recent progress in Rectified Flow (ReFlow) models provides a potential direction. Built upon Diffusion Transformer (DiT) backbones, ReFlow models substitute conventional U-Net convolutions with transformer-based sequence modeling, effectively capturing long-range dependencies across spatial and stylistic dimensions. This architectural design endows the model with exceptional in-context learning (ICL) capabilities 21 , facilitating rapid adaptation to multimodal conditioning signals. However, existing ICL-based image generation tasks 47 ; 33 typically rely on explicit exemplar pairs for guidance, making it challenging to decouple appropriate style patterns from a single style exemplar and integrate them with a target semantic prompt in ambiguous scenarios. In this paper, we propose a novel training-free style-guided text-to-image synthesis method that reformulates the problem as an in-context style learning task. Conditioned on textual semantic prompts, our method concatenates a reference style image with a masked target image to create a structured input, which is then fed into a pre-trained ReFlow-based inpainting model. This design enables the text prompt to provide high-level semantic and content guidance, while the style image supplies concrete visual exemplars of textures, colors, and artistic characteristics, facilitating spatially intuitive integration between style and content regions. However, this multimodal fusion introduces core technical difficulties, including imbalance and noise sensitivity in attention mechanisms. Concatenated attention often results in dominance by stronger activations, such as texture-rich styles overwhelming semantics, and suppression of the weaker branch. Noise introduced by the zero-mask in the inpainting pipeline further exacerbates this issue, leading to amplified perturbation across sampling steps. To address these problems and ensure coherent integration of semantic and stylistic signals, we introduce a semantic-style integration mechanism that dynamically reweights attention between textual and visual tokens, adaptively resolving conflicts between the guidance sources. The entire framework operates in a training-free manner, thereby enabling effective generalization to unseen styles. In summary, our key contributions are as follows: • We propose a novel training-free framework for style-guided image synthesis that seamlessly integrates textual semantic guidance with visual style exemplars, reformulating the task as an in-context style learning problem, supported by analyses of imbalance and noise sensitivity in multimodal attention fusion. • We introduce a dynamic semantic-style integration mechanism that adaptively balances cross-modal attention between text and style features, ensuring high fidelity to both semantic intent and stylistic characteristics. • Extensive experiments demonstrate that our method achieves flexible generalization to unseen styles and textual prompts without requiring model retraining, and can be applied to multiple style-guided image synthesis and various image editing tasks (see Fig. 2 ). 2 Related Work 2.1 Personalized Image Generation Diffusion models 50 ; 38 ; 40 have enabled remarkable progress in text-to-image generation by learning to reverse a noise-adding process and synthesizing high-quality images across diverse tasks 48 ; 46 ; 8 ; 57 ; 18 . Beyond generic generation, personalization has emerged as a key direction, aiming to generate images that preserve user-specific characteristics or follow user-defined visual styles 11 ; 39 ; 13 ; 24 . Our work focuses on style-guided image synthesis, a sub-task of personalized generation that aims to produce images consistent with the style of a given reference image. Early approaches, such as Textual Inversion 15 , learn a textual embedding from the input image to guide the generation process. DreamBooth 39 personalizes a diffusion model by fine-tuning it on a few images of a subject. The computational demands of DreamBooth’s full fine-tuning spurred the development of parameter-efficient methods like Low-Rank Adaptation (LoRA) 42 ; 43 . To further improve inference efficiency, some works 32 ; 4 ; 49 introduce a lightweight conditioning framework that unlocks precise control over large, frozen text-to-image diffusion