MultiRef: Controllable Image Generation with Multiple Visual References Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Irela
Ruoxi Chen, Dongping Chen †, Siyuan Wu, Sinan Wang, Shiyun Lang, Peter Sushko, Gaoyang Jiang, Yao Wan, Ranjay Krishna* · 2025
Read MultiRef: Controllable Image Generation with Multiple Visual References Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Irela by Ruoxi Chen, Dongping Chen †, Siyuan Wu, Sinan Wang, Shiyun Lang, Peter Sushko, Gaoyang Jiang, Yao Wan, Ranjay Krishna* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06905.txt.
Introduction
Ruoxi Chen and co-authors study image generators that take cues from several reference pictures at once, in a 2025 paper. Designers naturally mix elements from many sources, but most generators accept only a text prompt or a single image.
The team built a benchmark of nearly two thousand cases, half made by their own data engine and half drawn from the real world, all demanding a blend of multiple references. They also built a training set of thirty-eight thousand images. Testing three image-text models and six agent setups, they found even the best systems fell well short of the target answers. The gap, they argue, shows where more flexible creative tools must improve.
Catalog record
| CREATOR | Ruoxi Chen, Dongping Chen †, Siyuan Wu, Sinan Wang, Shiyun Lang, Peter Sushko, Gaoyang Jiang, Yao Wan, Ranjay Krishna* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MultiRef: Controllable Image Generation with Multiple Visual References Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3758292 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Artificial intelligence CCS: General and reference Evaluation Ruoxi Chen, Dongping Chen †, Siyuan Wu, Sinan Wang, Shiyun Lang, Peter Sushko, Gaoyang Jiang, Yao Wan, Ranjay Krishna* Abstract . Visual designers naturally draw inspiration from multiple visual references, combining diverse elements and aesthetic principles to create artwork. However, current image generative frameworks predominantly rely on single-source inputs - either text prompts or individual reference images. In this paper, we focus on the task of controllable image generation using multiple visual references. We introduce MultiRef-bench , a rigorous evaluation framework comprising 990 synthetic and 1,000 real-world samples that require incorporating visual content from multiple reference images. The synthetic samples are synthetically generated through our data engine RefBlend , with 10 reference types and 33 reference combinations. Based on RefBlend , we further construct a dataset MultiRef containing 38k high-quality images to facilitate further research. Our experiments across three interleaved image-text models ( i.e. , OmniGen, ACE, and Show-o) and six agentic frameworks ( e.g. , ChatDiT and LLM + SD) reveal that even state-of-the-art systems struggle with multi-reference conditioning, with the best model OmniGen achieving only 66.6% in synthetic samples and 79.0% in real-world cases on average compared to the golden answer. These findings provide valuable directions for developing more flexible and human-like creative tools that can effectively integrate multiple sources of visual inspiration. The dataset is publicly available at: https://multiref.github.io/ . † † cc-license: by 1 1 footnotetext: Corresponding author; † Project leader 1. Introduction Figure 1 . Image generation conditioned on multiple visual references provides more controllable and creative digital art generation than single image or textual reference. Digital artists and visual designers often create a new scene by blending elements from multiple source images: a color palette from a Monet painting, the architectural form of the Eiffel Tower from a photograph, and the texture from a hand-drawn sketch . Artists draw inspiration from multiple visual references, mixing diverse elements. This multi-reference creative process allows far more controllable image creation than relying on a single source of inspiration (Figure 1 ). However, current tools for this artistic process remain too primitive to be directly useful. Despite this artistic need, today’s image generators predominantly rely on single-source conditioning—either a text prompt ( i.e. , text-to-image ( Rombach et al. 2022 ; Esser et al. 2024 ) ) or one reference image ( i.e. , image editing ( Ruiz et al. 2023 ; Li et al. 2023a ) , image translation ( Wang et al. 2023 ; Han et al. 2024b ) ) at a time. In essence, asking a modern image generative model to “paint a scene in the style of Van Gogh with the composition of a photograph” requires specific prompt engineering ( Jia et al. 2024 ; Guo et al. 2025 ) or sequential editing ( Wang et al. 2024 ; Huang et al. 2024 ) . Moreover, visual references may have inconsistent viewpoints, styles, or semantics, and merging them can produce contradictions ( e.g. , blending a daytime landscape with a night-time style reference). Existing approaches like ControlNet ( Zhang et al. 2023 ) excel at following one conditioning signal ( i.e. , edge map and depth), but they are not inherently designed to handle multiple different conditions at once. Additionally, naively adding more control inputs usually confuses the model, leading to jumbled or degraded outputs ( Zhao et al. 2024 ) . Given these limitations in handling multiple references, there is a growing need to benchmark current multi-reference generation models. From our investigation, most popular benchmarks in generative modeling focus on text-to-image alignment or single-image editing. For example, IDEA-Bench ( Liang et al. 2024 ) targets professional design scenarios but still typically deals with one reference at a time or sequential editing. Similarly, ACE ( Han et al. 2024a ) evaluates alignment with instructions but does not stress-test combining several images. No established benchmark yet examines models on truly multi-reference tasks for their integrating complexity, making it hard to quantify current research progress. Figure 2 . An overview of RefBlend . It consists of reference generation (in yellow) and instruction prompt generation (in blue). First, various references (canny, depth, etc) are extracted from an original image. Then, a basic instruction prompt is formed from selected compatible references. Finally, the enhanced prompt is integrated with references to construct a sample. In this paper, we introduce MultiRef-bench , a benchmark that rigorously evaluates multi-reference generation models with 1,000 real-world samples and 990 synthetic samples that are programmatically generated. Specifically, we compile challenging user requests from Reddit ( Sushko et al. 2025 ) , where both references and ground truth images are real, to evaluate the image generation from multiple visual references. Our benchmark covers a spectrum of tasks, ranging from relatively straightforward scenarios—such as applying two independent references—to complex scenarios requiring simultaneous spatial and semantic alignment across multiple sources. To address the scarcity of multi-reference image generation datasets, we develop a novel synthetic data engine, termed RefBlend , that efficiently creates diverse training samples. RefBlend first extracts various visual references ( e.g. , depth maps, Canny edge, object masks) from original images using state-of-the-art extraction models. These references are then organized into a compatibility graph structure, where nodes represent individual references and edges indicate which references can be combined without contradictions, enabling diverse and high-quality multi-reference to image samples at scale. This engine can generate synthetic samples by flexibly combining diverse reference modalities— e.g. , a semantic map, human pose, and caption, each describing different aspects of the intended output—while treating the original image as the corresponding target. By controlling the data generation process, we automatically obtain rich ground-truth pairings of inputs and outputs. Finally, MultiRef-bench contains 990 synthetic samples covering 10 reference types and 33 reference combinations, far surpassing any existing collection in both scale and complexity. We propose new protocols to evaluate the generations using our benchmark. We leverage rule-based ( e.g. , MSE for depth) and model-based ( e.g. , ClipScore ( Hessel et al. 2021 ) for aesthetic) assessments for conditions that require precise evaluation ( e.g. , depth, mask and bbox) and fine-tuned MLLM-as-a-Judge ( Chen et al. 2024a ) for semantic-level assessments ( e.g. , caption, sketch and semantic map) in both reference-following and overall quality with human-annotated scores. We evaluate three interleaved image-text generation models ( e.g. , OmniGen ( Xiao et al. 2024 ) , ACE ( Han et al. 2024a ) , Show-o ( Xie et al. 2024 ) ) and 6 agentic frameworks ( e.g. , ChatDiT ( Huang et al. 2024 ) , LLM ( GeminiTeam 2023 ; Anthropic 2024 ) + Diffusion ( Rombach et al. 2022 ; Esser et al. 2024 ) ). Experimental results reveal that even the most advanced “general-purpose” image generators today struggle with multi-reference conditioning. State-of-the-art diffusion and autoregressive models that claim to support arbitrary conditioning ( e.g. , recent unified models) often falter when actually