Learning Complex Non-Rigid Image Edits from Multimodal Conditioning

Nikolai Warner, Jack Kolb, Meera Hahn, Jonathan Huang, Irfan Essa, Vighnesh Birodkar · 2024

Read Learning Complex Non-Rigid Image Edits from Multimodal Conditioning by Nikolai Warner, Jack Kolb, Meera Hahn, Jonathan Huang, Irfan Essa, Vighnesh Birodkar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10219.txt.

Introduction

Nikolai Warner and co-authors teach a system to place a person from one photo into a new scene with a new pose, in work from 2024. The edit is non rigid, since the body itself must bend and turn. Their method builds on a known image generation model and stays open to text and pose control.

Training needs pairs: a reference photo and a target photo of the same person in a new pose or place. Captions must state how the pose changed. The team built such a data set from action rich videos, with a multimodal model writing the captions. Keeping identity true in wild scenes proved the hard part.

Catalog record

CREATORNikolai Warner, Jack Kolb, Meera Hahn, Jonathan Huang, Irfan Essa, Vighnesh Birodkar
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Complex Non-Rigid Image Edits from Multimodal Conditioning Nikolai Warner, Jack Kolb, Meera Hahn, Jonathan Huang, Irfan Essa, Vighnesh Birodkar Abstract In this paper we focus on inserting a given human (specifically, a single image of a person) into a novel scene. Our method, which builds on top of Stable Diffusion, yields natural looking images while being highly controllable with text and pose. To accomplish this we need to train on pairs of images, the first a reference image with the person, the second a “target image” showing the same person (with a different pose and possibly in a different background). Additionally we require a text caption describing the new pose relative to that in the reference image. In this paper we present a novel dataset following this criteria, which we create using pairs of frames from human-centric and action-rich videos and employing a multimodal LLM to automatically summarize the difference in human pose for the text captions. We demonstrate that identity preservation is a more challenging task in scenes “in-the-wild”, and especially scenes where there is an interaction between persons and objects. Combining the weak supervision from noisy captions, with robust 2D pose improves the quality of person-object interactions. 1 Introduction Scene Ref Edit 1 Edit 2 Edit 3 Row 1: "He turns his body to the right and lowers his arms slightly." "He faces away showing his back side." "He raises his hand above his head." Row 2: "She raises her arms above her head." "She lowers her arms." "She lunges her right leg forward." Row 3: "He bends down reaching towards his toes." "He lowers his arms by his side." "He runs away with the ball." Figure 1 : Given a single image, multiple controllable identity-preserving edits can be specified with different text captions. Given a masked insertion scene and reference image containing a person to insert, our fine-tuned model inserts them into the scene controllable as controlled by a given text caption. Text and image-based inference on unseen images using an image + text model. Where captions are unavailable, we train the text and image model on a blank caption. Scene Ref Ours Ground Truth Row 1: "She bends forward at the waist, touching the ground with her hands spread apart." Row 2: "He goes from sitting on the ground to standing up." Row 3: "He follows through after swinging the baseball bat, moving it to his left side." Row 4: "He stands upright, looking down at the clothes in the basin." Figure 2 : Complex edits are achievable through weakly annotated supervision of a portion of the overall dataset, and finetuning jointly on text. Given a masked scene to insert a person into ("Scene") and a segmented crop of the person ("Ref"), the person is inserted into the scene. For comparison, we also provide the ground truth image that is paired with the caption and input. Given a scene, and person to insert, our model is capable of multiple non-rigid edits that preserve the identity of the person, despite a relatively small set of 13,487 weakly annotated captioned image pairs out of 78,000 videos. See Appendix Table 1 for dataset details and Section 3.3 for details on our weakly annotated captions. Generative diffusion models are an essential tool for creating high-quality images, videos, and 3D models. For applications involving human subject matter, the models can edit images in a controlled manner through textual and pose inputs. This capability allows for precise, user-defined modifications without the need for detailed manual editing. In this paper, we explore the use of generative diffusion models for conducting complex, non-rigid image edits that represent a challenging class of modifications. Despite their capabilities, current generative models do not fully leverage the extensive implicit world knowledge they accrue during training. By directing these models to manipulate this knowledge through structured edits explicitly, we aim to expand their practical applications in image editing. Our results show improvements in the model’s ability to preserve identity and contextual accuracy in generated images across controlled and uncontrolled environments. This work advances the capabilities of generative image editing, and lays groundwork for further developments in intuitive, user-centric image editing technologies. Prior work has organized the complexity of human subject image editing into rigid and non-rigid edits [ 16 ] . Rigid edits preserve the subject’s pose and alter their appearance (e.g., changing their shirt), while non-rigid edits modify the pose and may preserve appearance (e.g., generating yoga poses from a reference). Non-rigid edits are complex as they involve deforming the subject in a natural and coherent manner. Increasing the difficulty, non-rigid edits can include object interactions, requiring the model to know how to interact with the object’s class. For user applications, open-vocabulary text guidance is desirable for prompting edits. Central to our approach is the integration of multi-modal large language models (LLMs) with video data, enabling a novel form of noisy, self-supervised learning. This integration allows our models to learn text-controllable edits from less structured video data, approximating the unpredictability of real-world scenarios. Using video clips as datasets enriches the model’s understanding of human motion and interactions across various contexts, enhancing its capability to manage complex edits and improving its comprehension of human-object interactions within scenes. This paper contributes the following: 1. Controllable Non-Rigid Edits on Real World: We enable non-rigid edits of human subject scenes, controllable by text, on in-the-wild data and public datasets. We show considerable qualitative improvements over baseline models and algorithms. 2. Training with Language-Guided Controllability: We develop a new conditioning pipeline that uses multimodal LLMs to provide noisy guidance through world knowledge to condition and model non-rigid insertions. 3. Superior Person-Object Interactions : We improve controlling interactions between objects and persons. Our model integrates combinations of reference images, textual descriptions, and pose data, showing superior performance compared to baselines that are image-only, image-text, or image-pose. 4. Open-Source Annotated Dataset : We collect a human-annotated dataset of thousands of captions and processed video frames from multiple person-centric sources. We also validate prior work and re-train baselines that were originally trained on closed-source data. 2 Related Works Non-Diffusion Approaches Affordance-Diffusion Approaches Input Image MASACtrl ControlNet PIDM* Masked Input Reference Crop Our’s Kulal et al.† ↑ control prompt: “ He kicks his right leg up higher while leaning back. " ↑ control prompt: “ She extends her right leg back and arms outwards while ice skating. " ↑ control prompt: “ He lunges with his right leg. " ↑ control prompt: “ He runs towards the left. " ↑ control prompt: “ She holds her hands on her hips. " Figure 3 : Comparison of our approach to baselines for identity preservation and controllability of in-the-wild images. The input image is controlled using the prompts below each row, and the human subject is transferred to new frames for relevant models (our’s and Kulal et al.). No baseline achieves comparable identity-preserving non-rigid edits on in-the-wild data. Baselines either insert a person without controllability (Kulal et al.), or are controllable but fail to generalize to in-the-wild images (MASACtrl, PIDM). Our approach maintains similar photorealism to Kulal et al., with improved controllability. * PIDM works well on its training dataset but is brittle in the wild, likely due

Read the complete text