Vision-Language Binding in In-Context Image Generation
Chris Ge, Rohit Gandikota, Antonio Torralba, Tamar Rott Shaham · 2026
Read Vision-Language Binding in In-Context Image Generation by Chris Ge, Rohit Gandikota, Antonio Torralba, Tamar Rott Shaham (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24624.txt.
Introduction
Chris Ge and three colleagues show that in FLUX.2 in-context image generation, text tokens quietly absorb visual reference content and carry it into the output. Three causal interventions surface the binding: a text-to-image lens decoding text activations, attention knockout severing edges, and patching text activations between runs. Across 2,875 editing tasks, reference color, style, and scene setting travel through text tokens. Pixel-exact traits like a face or identity bypass text and flow directly image to image. Padding tokens of the text sequence host the binding.
Catalog record
| CREATOR | Chris Ge, Rohit Gandikota, Antonio Torralba, Tamar Rott Shaham |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60191 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24624
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision-Language Binding in In-Context Image Generation Chris Ge, Rohit Gandikota, Antonio Torralba, Tamar Rott Shaham Abstract In-context image generation models such as FLUX.2 [ 12 ] take a text prompt and an optional reference image as visual conditioning for the output. Internally, all three inputs --- text, reference image, and the noise tokens --- are concatenated and processed through a single attention stream, where all tokens can attend to one another. This leaves open how reference information flows through the model to produce the output image. We show that an implicit cross-modal binding emerges between the text tokens and the reference image: the text tokens absorb visual reference content during the forward pass, and that absorbed content causally influences the generated output. We surface this binding with three causal interventions on FLUX.2: T2I Lens, which decodes intermediate text-token activations through a text-to-image path; Attention Knockout, which severs specific attention edges; and I2I-to-I2I Patching, which copies text token activations between editing runs. Across 2,875 editing tasks on various images, including SUN397 and DreamBench++ datasets and images collected online, we observe a consistent division of labor: properties of the reference image, like color, style, and scene setting, are first written into the text tokens, which carry them to the generated image; pixel-exact properties like a specific face or instance identity bypass the text tokens and flow directly from reference to image through image-to-image attention. We further localize the reference-text binding to the padding tokens of the text sequence. These results show that text tokens in a multimodal DiT are not just prompt holders, but a structured channel for reference image content. More broadly, they suggest that even in unified-attention multimodal generative models, token modality structures how conditioning information is represented and routed across the network. 1 1 1 Our code and data are available at https://chrisg777.github.io/i2i-interp/ . 1 Introduction A single FLUX.2 forward pass can render a scene from a prompt, edit a reference image in response to an instruction, or stage a reference object into a completely new context. The same set of weights is used for all three. How does this happen, and what information is actually flowing between text tokens, reference image tokens, and generated image tokens inside the model when it does? In earlier image-to-image (I2I) diffusion models, this question does not arise by construction. Methods like Prompt-to-Prompt [ 8 ] and InstructPix2Pix [ 3 ] layered editing on top of a text-to-image U-Net [ 16 ] , with the reference image entering through a fixed conditioning channel and the instruction coming in through cross-attention. Text and image had distinct architectural roles, and their interaction was confined to specific layers one could point to directly. OmniGen [ 22 ] and later FLUX.1 Kontext [ 11 ] and FLUX.2 [ 12 ] changed this paradigm. Architecturally, they routed reference-image tokens into the same multimodal attention sequence as the text tokens, with no dedicated cross-attention layer for reference conditioning. They were also trained to also handle text-to-image (T2I) generation natively, in a single set of weights. The same multimodal attention block now switches, on a per-input basis, between a T2I forward pass with no reference and a reference-conditioned forward pass where reference content sits in the input sequence. This raises a question the architecture itself doesn’t answer: what role do the text tokens play when a reference image is included in the input? Are they still just prompt holders, as their text-to-image training would suggest, or do they pick up reference image content during the forward pass and serve as a medium that carries it into the generated image? An analogous question was recently posed for vision-language models that generate text from image plus text [ 9 ] ; we take it up for diffusion transformers that generate images from image plus text. We answer these questions with three causal interventions on the multimodal attention stream of FLUX.2, illustrated in Figure 1 . T2I Lens decodes the intermediate-layer text-token activations of a reference-conditioned forward pass through a text-to-image path, to read out what the text tokens encode at that point. Attention Knockout blocks specific attention edges, to test which pathways the model uses for each kind of reference content. I2I-to-I2I Patching copies text-token activations from one editing run into another to test which properties causally transfer with them. We find that the text tokens carry significantly more information about the reference image than their original training role as prompt holders would suggest. What they carry follows a clear pattern: properties that can be expressed in words are first written into the text tokens, which carry them to the generated image, while pixel-exact properties bypass the text tokens entirely and flow directly from reference to image. We further localize this binding to a specific region of the text sequence that has not, to our knowledge, been studied as a meaningful site of computation in this kind of model. Beyond FLUX.2, these findings have implications for unified-attention multimodal generative models more generally. A shared attention stream does not erase the modality of its tokens: each set continues to carry the kind of conditioning content best suited to its modality, even though no part of the training objective explicitly enforces this. The diagnostic tools we develop apply to any in-context multimodal model that mixes text and image tokens in a shared attention operation, and the routing patterns we identify can inform future design choices about which modality should carry which kind of conditioning. Figure 1 : Our three causal intervention methods on text tokens. (1) In T2I Lens , we patch the text residual stream after a given MM-DiT or DiT block into the raw text embeddings of a reference-free, unconditional text-to-image generation with an empty prompt , decoding the text-token activations at that point back into pixel space. (2) In Attention Knockout [ 5 ] , we mask out the attention paid by text tokens to reference tokens, or by image tokens to reference tokens, closing one possible pathway for reference information and isolating its causal effect. (3) In I2I-to-I2I Patching , we patch the same intermediate-layer text activations as T2I Lens from a source image-editing generation into a target image-editing generation that shares the same instruction prompt, testing whether the absorbed reference information transfers to the patched run’s output. 2 Related Work 2.1 Diffusion and Flow-Matching Transformers In FLUX.2, text tokens and image tokens share a unified attention operation through multimodal diffusion transformer (MM-DiT) blocks [ 4 ] . Concretely, FLUX.2 stacks double stream blocks , which are traditional MM-DiT blocks that maintain modality-specific QKV and MLP weights, followed by single stream blocks , which are closer to the original DiT [ 14 ] in applying a single shared set of weights to the combined token sequence. In both cases, there is no privileged cross-attention path between modalities, which is what makes the routing question we ask non-trivial: there is no single layer one can point at as the place where text and image meet. 2.2 In-Context Image Editing Beyond the methods discussed in the introduction, several DiT-based editors are relevant to our setup. ICEdit [ 24 ] and Group Relative Attention Guidance [ 23 ] both operate on the unified