TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Models

Teng-Fang Hsiao, Bo-Kai Ruan, Yi-Lun Wu, Tzu-Ling Lin, Hong-Han Shuai · 2025

Read TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Models by Teng-Fang Hsiao, Bo-Kai Ruan, Yi-Lun Wu, Tzu-Ling Lin, Hong-Han Shuai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15283.txt.

Introduction

Teng-Fang Hsiao and four co-authors present TF-TI2I, a 2025 method for generating images from text plus reference pictures, without any extra training. Existing image-and-text methods use only parts of the reference images or lose quality on complex multi-image instructions. TF-TI2I adapts modern text-to-image models such as SD3 by exploiting a quirk: text tokens in the model can implicitly pick up visual information from image tokens.

The method compresses each reference image into a compact visual summary and shares it selectively, using masking that limits context to instruction-relevant details. A winner-takes-all step picks the most relevant reference for each image token, taming distribution shift. The team also built FG-TI2I Bench, a benchmark for this task. Tests showed strong results on complex generation tasks.

Catalog record

CREATORTeng-Fang Hsiao, Bo-Kai Ruan, Yi-Lun Wu, Tzu-Ling Lin, Hong-Han Shuai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Models Teng-Fang Hsiao, Bo-Kai Ruan, Yi-Lun Wu, Tzu-Ling Lin, Hong-Han Shuai Abstract Text-and-Image-To-Image (TI2I), an extension of Text-To-Image (T2I), integrates image inputs with textual instructions to enhance image generation. Existing methods often partially utilize image inputs, focusing on specific elements like objects or styles, or they experience a decline in generation quality with complex, multi-image instructions. To overcome these challenges, we introduce Training-Free Text-and-Image-to-Image (TF-TI2I), which adapts cutting-edge T2I models such as SD3 without the need for additional training. Our method capitalizes on the MM-DiT architecture, in which we point out that textual tokens can implicitly learn visual information from vision tokens. We enhance this interaction by extracting a condensed visual representation from reference images, facilitating selective information sharing through Reference Contextual Masking—this technique confines the usage of contextual tokens to instruction-relevant visual information. Additionally, our Winner-Takes-All module mitigates distribution shifts by prioritizing the most pertinent references for each vision token. Addressing the gap in TI2I evaluation, we also introduce the FG-TI2I Bench, a comprehensive benchmark tailored for TI2I and compatible with existing T2I methods. Our approach shows robust performance across various benchmarks, confirming its effectiveness in handling complex image-generation tasks. Our project page: https://bluedyee.github.io/TF-TI2I_page/ Figure 1 : Illustration of our proposed TF-TI2I, designed to leverage multiple image references for generation and editing without additional training. The importance of image references is illustrated in the middle part, where excluding any reference significantly alters the output. 1 Introduction “Don’t tell me the moon is shining; show me the glint of light on broken glass.” — Anton Chekhov (1860-1904) Recent advances in text-to-image (T2I) generation empower users to produce compelling visual artwork using only textual prompts [ 44 , 42 , 57 , 27 , 36 ] . However, text descriptions alone are insufficient for conveying the nuanced spatial relationships, specific object characteristics, and stylistic details necessary to fulfill users’ intentions. To address this, various approaches have integrated additional visual guidance into T2I pipelines, leveraging conditional-control methods [ 61 , 63 , 32 ] , inpainting techniques [ 25 , 64 , 58 , 24 ] , style transfer [ 13 , 23 , 15 , 19 ] , and customized image synthesis [ 46 , 11 , 18 , 60 , 54 ] . Despite their effectiveness, current methods are typically constrained to a single functionality. For example, conditional-control, inpainting, and style-transfer approaches focus on one aspect of visual guidance. Similarly, customization methods excel at object-based references but struggle with other factors e.g., texture or background. One solution is the use of multimodal large language models (MLLMs) in image generation [ 40 , 50 ] , which can handle both visual and textual inputs. However, MLLM-based generation still lags behind specialized T2I models in terms of output quality and resolution, largely due to the limited availability of text-and-image-to-image training data. For instance, KOSMOS-G [ 40 ] , utilizing BLIP2 [ 29 ] and CLIP-Seg [ 29 ] to autonomously identify objects as image inputs, remains restricted to object-level conditioning without providing finer control over texture or background details. To enhance performance without larger training data, we explore MM-DiT—a fully transformer-based multimodal architecture used in state-of-the-art T2I models such as SD3 [ 17 ] and FLUX [ 27 ] . We contend that MM-DiT’s multimodal attention demonstrates cross-modal understanding [ 51 , 20 , 16 ] , which allows textual tokens to naturally integrate implicit visual information from image latent during inference. Unlike methods that require additional training to establish image-reference correlations, we propose Training-Free Text-and-Image-to-Image (TF-TI2I) by (1) harnessing the inherent ability of MM-DiT to infuse textual tokens with implicit visual context during generation, and (2) enabling these tokens to be shared across different inputs as contextual tokens. As our approach relies only on the model’s intrinsic properties, we can extend T2I models with TI2I capabilities without fine-tuning. However, challenges arise as the number of image references increases. Firstly, mutual interference between different references can lead to unintended visual blending. For instance, as shown in Fig. 1 , when generating a background following reference 4 (Starry Night), elements from the backgrounds of references 1, 2, and 3 might inadvertently be incorporated due to insufficient differentiation among the references. To address the mutual interference among multiple references, we propose References Contextual Masking (RCM) . This technique limits the visual information learned from contextual tokens by focusing exclusively on vision tokens that are more related to the given instructions. This ensures that only the relevant visual information of a specified reference is utilized in the image generation process. By doing so, RCM effectively reduces the undesired blending of visual features from different references, maintaining clarity and fidelity to the original input conditions. Secondly, as a training-free approach, TF-TI2I is also susceptible to distribution shifts that can degrade generation quality, a challenge noted in related techniques [ 13 , 35 , 15 ] . To counteract this, we introduce the Winner-Takes-All (WTA) module. This module assigns each vision token exclusively to one reference at a time, ensuring that each token robustly represents the visual characteristics of its assigned references. Concurrently, the WTA module facilitates the incorporation of visual information from other references through attention between the visual tokens. This dual mechanism not only maintains each reference’s integrity within the output but also enriches the visual details, resulting in high-quality images that adhere to both intended references and textual instruction. Finally, recognizing the need for a comprehensive evaluation framework for both TI2I models [ 50 , 40 ] and visually guided T2I models [ 56 , 3 , 22 , 61 , 8 ] , we introduce the Fine-Grained TI2I Benchmark (FG-TI2I Bench) tailored for general image generation scenarios. Drawing inspiration from EditBench [ 53 ] , which assesses detailed object attributes like material, color, and shape, our benchmark structures prompt instructions around four key components and encompass 6 fundamental TI2I sub-tasks. These components are described using either text or images, making our framework compatible with visually guided T2I methods. Our approach achieves state-of-the-art performance on 12 out of 18 evaluation metrics across general-purpose TI2I tasks and remains highly competitive in task-specific benchmarks such as DreamBench [ 46 ] for customization and Wild-TI2I [ 52 ] for editing scenarios. Our contributions are summarized as follows: • We discover the implicit-context learning capability of MM-DiT and design TF-TI2I, a training-free approach, to generate images from different visual references and text prompts. This further augments the existing T2I model with TI2I capability. • We develop Reference Contextual Masking and Winner Takes All modules to mitigate multi-reference problems, leading to more visually satisfactory results. • We introduce FG-TI2I Bench, a comprehensive benchmark designed to evaluate a variety of visually guided T2I application e.g . customization and style transfer. Our approach achieves state-of-the-art results in 12 out of 18 evaluation metrics across FG-TI2I and also remains competitive in DreamBench [ 46 ] and Wild-TI2I [ 52 ] . 2 Related Work 2.1 TI2I with single aspect of reference To expand the creativity of pretrained T2I models with TI2I capability, finetuning-based approaches [ 46 , 60 , 19 , 18 , 12 , 41 ] , such as DreamBooth [

Read the complete text