Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting

Xinyue Pan, Yuhao Chen, Fengqing Zhu · 2026

Read Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting by Xinyue Pan, Yuhao Chen, Fengqing Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17666.txt.

Introduction

Xinyue Pan, Yuhao Chen, and Fengqing Zhu generate multi-food images without training. Real meal photos hold many food items at once. Dietary assessment uses such images to judge intake. Recipe tools use them to show finished dishes. Diffusion models struggle with multi-food scenes. Adjacent foods blend into each other, a flaw called object entanglement.

Their training-free method uses prompt grafting. Grafting joins prompts so each food keeps its place. Clean multi-food images can augment scarce datasets. That helps diet apps learn from fewer real photos. The work targets a niche where errors are easy to spot. A plate with merged foods fools no one.

Catalog record

CREATORXinyue Pan, Yuhao Chen, Fengqing Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17666

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting Xinyue Pan, Yuhao Chen, Fengqing Zhu Abstract Real-world meal images often contain multiple food items, making reliable compositional food image generation important for applications such as image-based dietary assessment, where multi-food data augmentation is needed, and recipe visualization. However, modern text-to-image diffusion models struggle to generate accurate multi-food images due to object entanglement, where adjacent foods (e.g., rice and soup) fuse together because many foods do not have clear boundaries. To address this challenge, we introduce Prompt Grafting (PG) , a training-free framework that combines explicit spatial cues in text with implicit layout guidance during sampling. PG runs a two-stage process where a layout prompt first establishes distinct regions and the target prompt is grafted once layout formation stabilizes. The framework enables food entanglement control: users can specify which food items should remain separated or be intentionally mixed by editing the arrangement of layouts. Across two food datasets, our method significantly improves the presence of target objects and provides qualitative evidence of controllable separation. I Introduction Compositional food image generation is crucial for many real-world applications. In image-based dietary assessment, for example, a single eating occasion often contains multiple foods [ 27 , 7 , 5 ] . To train a reliable food image analysis model, it often needs large amounts of diverse multi-food images. Beyond dietary assessment, the task also enables personalized recipe visualization, menu design, and food-related marketing, where clear description of multiple food items are usually required [ 24 , 28 , 29 , 31 ] . Although recent work on compositional image generation have achieved remarkable progress [ 18 , 4 , 1 , 23 ] , they still struggle with the object entanglement problem. This issue is common in the food domain because many food items lack clear geometric boundaries, causing adjacent foods to merge into a single, indistinct region. For example, white rice and beef noodle have soft textures and irregular shapes, as shown in Figure 1 . Even the recent Stable Diffusion 3 (SD3) model [ 3 ] entangles two food objects that should appear on separate plates. Fig. 1: Example compositional food images generated by stable diffusion v3 model (SD3) and our method with corresponding reference images. Recent text-to-image compositional generation work primarily builds upon Stable Diffusion v1 (SD1) [ 25 ] using only text prompts as input [ 18 , 1 , 4 , 23 ] . However, the CLIP text encoder in SD1 [ 21 ] often attends to only one or two tokens, resulting in frequent object omissions [ 10 , 35 ] . Newer models such as SD3 [ 3 ] and FLUX.1 [ 13 , 14 ] mitigate this issue through dual encoders (CLIP [ 21 ] +T5 [ 22 ] ), yet they still exhibit a strong single-object bias inherited from pretraining on LAION [ 26 ] , where most images contain one centrally placed object. As shown in Section III-B , SD3 still fails to initialize correct multi-object layouts during early denoising steps of image generation, leading to object entanglement. To address this problem, we propose a framework built upon Prompt Grafting (PG). Before introducing PG, we acknowledge that adding simple spatial cues to the prompt (e.g., “on the lower left,” “on the top right”) can sometimes help encourage foods to appear in different regions, but this alone is unreliable, as shown in Section III-C . Therefore, we propose Prompt Grafting (PG) that initializes a separable layout and then places food items into layouts. In early denoising, we condition on a layout prompt to generate distinct layouts; once layout formation stabilizes, we graft to the target prompt so that content fills those layouts. Overall, this method ensures adjacent foods remain visually separate. Crucially, our method is training-free (no finetuning beyond pretrained model) and requires no annotated layouts. Additionally, it enables food entanglement control: users simply place the items they wish to mix on the same layout. We validate the approach on two food datasets with both quantitative and qualitative ways: detection-based metrics improve markedly over SD3 baselines; we also show qualitative generalization to non-food domain. Our contributions can be summarized as follows: • To the best of our knowledge, this is the first work to address object entanglement in compositional food image generation. • We present empirical analysis that reveal both object‑omission and object‑entanglement issues in compositional food image generation. • We introduce Prompt Grafting (PG) , a training-free framework that disentangles overlapping objects and mitigates entanglement. II Related Work II-A Food Image Generation High-quality food images are essential for training image analysis models in tasks like classification, segmentation, and portion estimation. Some works tackled data scarcity by synthesizing food images: Fu et al. exposed inter-class confusion in StyleGAN-generated food and showed that isolating training to one category improves visual fidelity [ 11 ] . Han et al. compared diffusion models against GANs for food synthesis, finding that diffusion approaches better handle the wide stylistic variations in cooking [ 6 ] . Yamamoto et al. enhanced semantic alignment by injecting CLIP embeddings into the generative process [ 32 ] . Li et al. ’s ChefFusion couples a Transformer-based recipe generator with a diffusion backbone to jointly produce cooking instructions and images [ 16 ] , while Ma et al. ’s MLA-Diff integrates ingredient–image CLIP features via an attention-fusion module for more realistic renderings [ 19 ] . Most recently, Yu et al. ’s CW-Food framework separates shared class-level traits from individual-instance details using a Transformer fusion layer and LoRA fine-tuning, enabling diverse food synthesis from text labels alone [ 34 ] . Despite these advances in single food and recipe-guided generation, none directly address the unique challenges of composing multiple distinct food items into one coherent form. II-B Compositional Image Generation Compositional image generation aims to generate an image containing multiple, specified objects. Two core challenges arise: (1) attribute binding , ensuring attributes (e.g., color, size) correctly attach to their objects; and (2) object existence , preventing the omission of required items. There are some related works that have used Stable Diffusion v1 model (SD) in this task by manipulating attention maps during the denoising process. Liu et al. dispatch separate diffusion streams for different prompt components, improving token–object alignment [ 18 ] . Feng et al. proposed Structured Diffusion , which parses prompts into paired entities (e.g., “red car,” “white house”) and applies structured guidance to enforce correct attribute–object pairings [ 4 ] . Chefer et al. introduce an Attend and Excite pipeline that increases weights on low‑attention tokens to reduce object omission [ 1 ] . Rassin et al. proposed Synegn , which leverages syntactic parsing and a custom loss to align related words and prevent incorrect bindings [ 23 ] . Despite their success, all of these methods are built on SD’s CLIP text encoder, which is limited to simple prompts, and none of them address the object entanglement problem in image generation. Several methods introduce additional modules to achieve compositional control, such as LayoutDiffusion [ 36 ] , GLIGEN [ 17 ] , and Instance Diffusion [ 30 ] . While effective, these approaches require architectural extensions, more GPU resources, and additional inputs

Read the complete text