Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation
Bolin Lai, Felix Juefei-Xu, Miao Liu, Xiaoliang Dai, Nikhil Mehta, Chenguang Zhu, Zeyi Huang, James M. Rehg, Sangmin Lee, Ning Zhang, Tong Xiao · 2024
Read Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation by Bolin Lai, Felix Juefei-Xu, Miao Liu, Xiaoliang Dai, Nikhil Mehta, Chenguang Zhu, Zeyi Huang, James M. Rehg, Sangmin Lee, Ning Zhang, Tong Xiao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01027.txt.
Introduction
Bolin Lai and co-authors build a model that learns a new photo edit from just a few examples plus a text hint. Their 2024 model answers a limit of diffusion tools, which reason poorly from visual examples. Text alone can be vague for rare or hard-to-describe edits.
A group attention design splits the job into two stages: first learn the edit, then apply it. A second rule keeps the learned change apart from stray content in the example photos. In tests, the model beat past few-shot editors by a wide margin in human ratings. More examples, or more varied ones, lift results further. The authors frame the design as fast, on-the-spot learning of edits.
Catalog record
| CREATOR | Bolin Lai, Felix Juefei-Xu, Miao Liu, Xiaoliang Dai, Nikhil Mehta, Chenguang Zhu, Zeyi Huang, James M. Rehg, Sangmin Lee, Ning Zhang, Tong Xiao |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01027
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation Bolin Lai, Felix Juefei-Xu, Miao Liu, Xiaoliang Dai, Nikhil Mehta, Chenguang Zhu, Zeyi Huang, James M. Rehg, Sangmin Lee, Ning Zhang, Tong Xiao Abstract Text-guided image manipulation has experienced notable advancement in recent years. In order to mitigate linguistic ambiguity, few-shot learning with visual examples has been applied for instructions that are underrepresented in the training set, or difficult to describe purely in language. However, learning from visual prompts requires strong reasoning capability, which diffusion models are struggling with. To address this issue, we introduce a novel multi-modal autoregressive model, dubbed InstaManip , that can insta ntly learn a new image manip ulation operation from textual and visual guidance via in-context learning, and apply it to new query images. Specifically, we propose an innovative group self-attention mechanism to break down the in-context learning process into two separate stages – learning and applying, which simplifies the complex problem into two easier tasks. We also introduce a relation regularization method to further disentangle image transformation features from irrelevant contents in exemplar images. Extensive experiments suggest that our method surpasses previous few-shot image manipulation models by a notable margin ( ≥ \geq 19% in human evaluation). We also find our model can be further boosted by increasing the number or diversity of exemplar images. Please check out our project page ( https://bolinlai.github.io/projects/InstaManip/ ). Figure 1 : When learning a new image manipulation operation that is unseen in the training set (as shown above), textual instructions directly point out the subject and provide high-level semantic guidance, while exemplar images mitigate linguistic ambiguity and show more local details that are difficult to describe in language. Our proposed multi-modal autoregressive model – InstaManip takes advantage of both textual and visual guidance to learn a representation of the desired transformation, and applies it to a new query image. † † footnotetext: † This work was done during Bolin’s internship at GenAI, Meta. 1 Introduction The recent emergence and advancement of diffusion models have greatly facilitated the boom of text-to-image generation [ 13 , 54 , 52 , 53 , 3 , 55 , 9 , 48 , 10 , 44 ] , which has further driven a remarkable development in text-guided image manipulation [ 7 , 57 , 40 , 23 , 42 , 83 ] . However, existing models still suffer from a notable performance drop when the manipulation is difficult to articulate textually or when instructions deviate from the training data [ 7 , 43 ] . For example, when we want to turn a plain car to a Lamborghini, the model may fail to correctly understand the shape and texture only from the word “Lamborghini”, if it is not included in training data ( Fig. 2 (a)). It is also hard for humans to accurately describe all details of Lamborghini in texts. Moreover, we are living in a world where new concepts constantly emerge across the Internet and social media, which are rarely covered in any training set. The generalization limitation hinders existing models from being applied in the real world. A straightforward solution to this problem is additionally providing a few exemplar images for the model ( i.e . , few-shot image manipulation as shown in Figs. 1 and 2 (b)), which has been studied in some recent work [ 75 , 62 , 43 , 86 ] . All of these methods rely on the architectures of diffusion model [ 54 ] and ControlNet [ 84 ] . However, learning from visual examples requires a strong reasoning capability to separate image-to-image transformation features from the irrelevant content in exemplar images. Diffusion models are excellent in generation, yet still weak in reasoning [ 65 ] . In contrast, autoregressive architectures, especially large language models (LLMs), have shown remarkable reasoning performance, which enables them to learn new tasks from prompts without finetuning ( i.e . , in-context learning) [ 2 , 76 , 11 , 87 , 45 ] . In this paper, we make an attempt to address few-shot image manipulation problem by harnessing the in-context learning feature of autoregressive models, specifically multi-modal large language models (MLLMs). Prior to our work, many efforts have been made to turn an autoregressive architecture into a generalist model that can handle various visual tasks [ 25 , 61 , 20 , 56 , 2 , 38 , 22 , 17 , 82 , 74 ] , such as visual question answering, image completion, and semantic segmentation. However, in-context learning for few-shot image manipulation with autoregressive models is still an understudied problem. In addition, few-shot image manipulation essentially consists of two stages: (1) learning the desired transformation from textual guidance and visual examples, and then (2) applying learned knowledge to a new query image (which has also been shown in human’s learning process [ 67 , 64 ] ). Most existing autoregressive models combine the two stages in a single step while applying in-context learning, and fully rely on self-attention to automatically model the dependence across given examples, query images and desired output. These straightforward approaches increase the problem complexity, which leads to a bottleneck in learning the desired manipulation rules and transferring to other images. Figure 2 : Comparison of InstructPix2Pix [ 7 ] and our model. We exclude “Lamborghini” from training set for both models. To address these issues in few-shot image manipulation, we introduce InstaManip , an innovative multi-modal autoregressive architecture that models the two stages separately. Specifically, we propose a novel group self-attention mechanism, which disentangles the learning and applying stages by splitting the input prompt into two groups and conducting self-attention in each group separately, exactly aligned with aforementioned human’s cognition. Furthermore, we introduce a relation regularization scheme to encourage instances with similar manipulation to be encoded close to each other, which drives the model to distinguish the desired manipulation features from irrelevant image contents. The experiments show that the proposed method achieves new state-of-the-art performance when applied to unseen image manipulation instructions. Overall, our contributions can be summarized as follows: • We introduce InstaManip, a novel autoregressive model that unleashes in-context learning capability of MLLMs for few-shot image manipulation. • We propose the innovative group self-attention method that breaks down in-context learning into two stages – learning and applying, following human’s learning process. We also propose a relation regularization strategy to further separate underlying transformation rules from undesired visual features. • Extensive experiments suggest that our proposed method prominently improves the in-context learning capability and outperforms existing few-shot image manipulation models. Our model is further improved by using more examples or increasing the diversity of visual prompts. 2 Related Work Few-shot Image Manipulation. Text-guided image manipulation has been widely studied since the emergence of diffusion models [ 7 , 57 , 40 , 23 , 42 , 47 , 70 , 12 , 46 , 41 , 15 , 34 , 6 , 30 , 29 ] . Recently, few-shot learning is adopted to this problem for a better performance by using one or more exemplar image pairs as reference [ 33 , 58 ] . Sun et al . [ 62 ] propose ImageBrush, which frames query image and a