AM-Adapter: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild
Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, Seungryong Kim · 2024
Read AM-Adapter: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild by Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, Seungryong Kim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03150.txt.
Introduction
Siyoon Jin and co-authors build a tool that paints new images with the look of a sample photo. Their 2024 paper tackles exemplar-based synthesis: the layout comes from a semantic map, the style from one scene photo. Older tools leaned on text prompts and could not use a photo as style input. Newer ones falter on busy scenes with many objects.
Their adapter matches local looks across images inside the attention layers of a diffusion model. Semantic maps guide the matching, and users can assign object details to chosen spots. Training runs in stages to keep making and matching apart. At test time, an automatic search picks good sample pairs. With few added parameters, the tool leads on both layout fit and look fidelity.
Catalog record
| CREATOR | Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, Seungryong Kim |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03150
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AM-Adapter: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, Seungryong Kim Abstract Exemplar-based semantic image synthesis generates images aligned with semantic content while preserving the appearance of an exemplar. Conventional structure-guidance models like ControlNet, are limited as they rely solely on text prompts to control appearance and cannot utilize exemplar images as input. Recent tuning-free approaches address this by transferring local appearance via implicit cross-image matching in the augmented self-attention mechanism of pre-trained diffusion models. However, prior works are often restricted to single-object cases or foreground object appearance transfer, struggling with complex scenes involving multiple objects. To overcome this, we propose AM-Adapter ( A ppearance M atching Adapter ) to address exemplar-based semantic image synthesis in-the-wild, enabling multi-object appearance transfer from a single scene-level image. AM-Adapter automatically transfers local appearances from the scene-level input. AM-Adapter alternatively provides controllability to map user-defined object details to specific locations in the synthesized images. Our learnable framework enhances cross-image matching within augmented self-attention by integrating semantic information from segmentation maps. To disentangle generation and matching, we adopt stage-wise training. We first train the structure-guidance and generation networks, followed by training the matching adapter while keeping the others frozen. During inference, we introduce an automated exemplar retrieval method for selecting exemplar image-segmentation pairs efficiently. Despite utilizing minimal learnable parameters, AM-Adapter achieves state-of-the-art performance, excelling in both semantic alignment and local appearance fidelity. Extensive ablations validate our design choices. Code and weights will be released.: https://cvlab-kaist.github.io/AM-Adapter/ Figure 1 : AM-Adapter enables Exemplar-based Semantic Image Synthesis in-the-Wild. (a) Given an exemplar image and a target segmentation, AM-Adapter generates high-quality images that retain the local appearance of the exemplar and the accurate image structure defined by the segmentation map. We demonstrate the versatility of our method in various applications, including (b) controllable one-to-one appearance transfer with user-defined guidance, (c) image-to-image translation and (d) segmentation-based image editing. 1 Introduction Figure 2 : Controllability of our AM-Adapter: The M -to- N setting refers to a multiple-object many-to-many transfer, where M and N denote the number of instances of a specific category in the exemplar and target, respectively. By default, as shown in (a), AM-Adapter automatically matches appearance based on structural similarity. In addition, as shown in (b), it allows user-defined guidance to precisely transfer specific objects (e.g., red car) to target instances (White-masked objects indicate user-specified matches). Exemplar-based semantic image synthesis [ 44 , 1 , 25 , 19 ] aims to generate an image aligned with given semantic content (e.g., segmentation maps) while faithfully reflecting user-defined visual attributes (e.g., styles, lighting, or object details). This task is critical for various downstream applications, including autonomous driving [ 6 ] , medical imaging [ 49 ] , and virtual/augmented reality (AR/VR) [ 8 ] . Recent advances in structure-guidance methods [ 47 , 32 , 16 ] such as ControlNet allow fine-grained spatial control but rely on text prompts (e.g., ‘realistic’ or ‘nighttime’) for appearance. As shown in Figure 7 , they often fail to transfer exemplar-specific details, as text alone is insufficient to convey fine-grained visual information. To address this, several methods have utilized exemplars for visual conditioning [ 44 , 1 , 19 ] , leveraging pre-trained image encoders such as CLIP [ 35 ] . This is achieved by mapping the exemplar’s features from the encoder onto the pre-trained diffusion model using linear projection heads [ 44 , 18 , 7 , 42 , 33 ] . However, as shown in Figure 7 , these models still struggle to accurately transfer local appearances from exemplars, as global mappings via linear layers are insufficient for embedding complex visual features. As a result, they are often limited to global appearance transfer while failing to capture fine-grained, object-specific details. To resolve these, some studies [ 1 , 19 , 2 , 28 ] have explored local appearance transfer by hand-crafted attention control in self-attention modules of pre-trained diffusion models. Prior works [ 2 , 28 ] achieve this through augmented self-attention, providing the key and value matrices of the exemplar to the self-attention that synthesize an image aligned with the given semantic content. This can be seen as finding correspondences between synthesized and exemplar images via implicit query-key matching, then warping appearance using the corresponding value matrices. While effective in constrained settings, they struggle in complex, real-world scenes with multiple objects and geometric variations. As shown in Figure 7 , implicit matching often misaligns categories (e.g., cars to buildings, trees to roads), causing artifacts and distortions. Recent methods introduce additional constraints [ 28 , 1 , 19 ] , yet remain limited to single or foreground object cases, making them ineffective for multi-object many-to-many scenarios. The most straightforward solution is to fine-tune the augmented self-attention module to improve matching. However, it is challenging as the module needs to learn both matching and generation simultaneously, leading to an unstable training signal and overfitting, as discussed in Figure 12 (d). While recent studies [ 12 ] attempted to fine-tune such model, they are typically limited to specific domains, such as foreground-centric human videos [ 13 ] . Importantly, the aforementioned methods fail to capture object-specific styles, as they either apply style globally, ignoring local details, or focus on single/foreground object. We define this task as exemplar-based semantic image synthesis in-the-wild , extending appearance transfer to complex, scene-level multi-object transfer. To solve this, we propose AM-Adapter , which enhances implicit matching in augmented self-attention of pre-trained diffusion by integrating semantics from segmentation maps. This enables both global and local appearance transfer through semantic correspondence and structural similarity, while preserving scene composition and ensuring natural object-wise appearance transfer. Our architecture consists of two branches: Appearance Net , which extracts local appearance from an exemplar, and Structure Net , which synthesizes a target image from random noise, aligned with a target segmentation map. To harness the semantic relationship between exemplar and target images for improved matching, we first construct a categorical matching cost between the exemplar and target segmentation maps, which is then concatenated with the implicit matching cost in self-attention. To find locally consistent matches within the 4D space between the exemplar and target, we refine the combined cost using a 4D cost aggregation module. The final output is added residually to the original implicit matching cost for stable training. To further disentangle generation and matching, we adopt a stage-wise training approach. First, we pre-train the diffusion model and ControlNeXt [ 32 ] on the target domain, and then train the matching adapter while keeping the diffusion model and ControlNeXt frozen. This approach is highly efficient, as it only requires updating the matching adapter’s parameters, preserving the image quality and structural consistency of the pre-trained models. During inference, previous studies [ 19 , 1 ] manually select exemplar images, which may be labor-intensive in a specific setting. To address this, we propose an automatic exemplar retrieval method that selects exemplar image-segmentation pairs from a large pool to maximize matchable regions with the synthesized target image. Despite the limited number of learnable parameters, our method achieves state-of-the-art performance in complex segmentation-to-image tasks, such as driving scenes [ 45 , 4 ]