InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization

Daniel Gilo, Or Litany, [0.3em] Technion -- Israel Institute of Technology · 2025

Read InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization by Daniel Gilo, Or Litany, [0.3em] Technion -- Israel Institute of Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14899.txt.

Introduction

Daniel Gilo and Or Litany address editing a scene consistently from only a few input views. Given sparse photos and a text instruction, the edit must agree across all viewpoints. Past methods based on neural fields or temporal attention often create artifacts here. Their framework distills a two dimensional diffusion editor into a pretrained multi view diffusion model.

The multi view student replaces the usual neural field consolidator in score distillation. New adaptations include incremental student updates, a teacher noise schedule that prevents degeneration, and an attention change for consistency. The data driven three dimensional prior helps keep edits coherent. The method targets sparse view editing where geometry cues are limited but consistency is essential.

Catalog record

CREATORDaniel Gilo, Or Litany, [0.3em] Technion -- Israel Institute of Technology
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-14899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization Daniel Gilo, Or Litany, [0.3em] Technion -- Israel Institute of Technology Abstract We address the task of multi-view image editing from sparse input views, where the inputs can be seen as a mix of images capturing the scene from different viewpoints. The goal is to modify the scene according to a textual instruction while preserving consistency across all views. Existing methods, based on per-scene neural fields or temporal attention mechanisms, struggle in this setting, often producing artifacts and incoherent edits. We propose InstructMix2Mix (I-Mix2Mix), a framework that distills the editing capabilities of a 2D diffusion model into a pretrained multi-view diffusion model, leveraging its data-driven 3D prior for cross-view consistency. A key contribution is replacing the conventional neural field consolidator in Score Distillation Sampling (SDS) with a multi-view diffusion student, which requires novel adaptations: incremental student updates across timesteps, a specialized teacher noise scheduler to prevent degeneration, and an attention modification that enhances cross-view coherence without additional cost. Experiments demonstrate that I-Mix2Mix significantly improves multi-view consistency while maintaining high per-frame edit quality. Additional visualizations and code are available on our project page . 1 Introduction Multi-view image editing seeks to modify a scene captured from multiple viewpoints while preserving consistency across views. Typical edits include texture and color changes, semantic manipulations, or geometric transformations, with applications in product imagery, real-estate and interior design visualization, AR/VR, and cinematic post-production. However, multi-view editing is a difficult task that traditionally requires skilled artists, making automation highly desirable. In recent years, a variety of methods have been proposed for editing 3D scenes. Due to the difficulty of producing supervision in the form of high-quality paired scenes before and after editing, most of these approaches avoid direct multi-view editing, instead leveraging monocular editors such as InstructPix2Pix 7 , either iteratively 22 ; 61 or through distillation 39 ; 31 . These approaches achieve 3D-consistent edits by operating on dense scene representations like NeRFs 46 or 3D Gaussian Splats 32 . However, in many real-world scenarios, users have access only to a sparse set of views—such as a few casually captured photos or product images—which provide limited scene coverage and challenge existing 3D editing methods to maintain geometric and appearance consistency. In this work, we tackle the challenging problem of multi-view editing from sparse input images (a mix of images). Given a few source views and a textual editing instruction, our aim is to generate edits that faithfully follow the prompt while remaining consistent across all viewpoints. Following prior work, we leverage powerful 2D editors and lift their capabilities to 3D using Score Distillation Sampling (SDS) 50 . However, we propose a new approach to this paradigm. We observe that current approaches face inherent limitations. Neural field representations are trained per scene – they do not hold 3D prior in their network weights. Instead, they achieve 3D consistency by incorporating a physical prior through the rendering equations. A dense set of input images is required, however, to transform this prior into an effective consolidator. To achieve consistency with sparse views, we instead propose to incorporate a consolidator that embeds a strong, data-driven 3D prior directly in its weights: a multi-view synthesis diffusion model. While such models are trained to generate view-consistent scenes (e.g., from text or images), they lack editing capabilities. We bridge this gap by combining the strengths of both paradigms—distilling edits from a 2D editor (the teacher) into the multi-view model (the student). Concretely, we use InstructPix2Pix as the teacher and Stable Virtual Camera (SEVA) 76 as the student. By leveraging a student model with an inherent 3D prior, our method — I-Mix2Mix — produces robust, geometrically coherent, and visually consistent edits even from extremely sparse inputs. Replacing the neural field with a multi-view diffusion model within the SDS framework is not straightforward, and requires careful adaptation of several key steps. Instead of rendering from a scene representation, we sample from the diffusion model; to avoid costly full trajectories, we distill incrementally across student timesteps. We also introduce a specialized teacher noise scheduler to prevent collapse to poor local minima and an attention modification that strengthens multi-view consistency without extra cost. Together, these components yield a framework for consistent multi-view editing, producing high-quality results even with very sparse inputs. To summarize, our contributions are: 1. We present I-Mix2Mix, a novel framework for distilling the knowledge of a powerful monocular editor into a pretrained multi-view diffusion model, leveraging its data-driven 3D consistency. 2. Through careful consideration of the SDS key steps, we introduce novel adaptations to support personalization of our multi-view student. 3. We demonstrate that this approach produces high-quality, consistent multi-view edits, effectively extending the SDS framework to scenarios with limited viewpoints. We evaluate I-Mix2Mix against popular multi-view editing methods, demonstrating significant improvements in cross-view consistency both qualitatively and quantitatively in the sparse-view setting. At the same time, our method maintains competitive per-frame editing performance, highlighting the practical benefits of leveraging a data-driven multi-view prior within the SDS framework. 2 Related Works 3D editing. Editing 3D scenes or objects typically assumes a pre-optimized model such as a NeRF 46 or 3DGS 32 . Early works explored direct NeRF manipulation via scribbles 41 , sketches 45 , reference images 4 , meshes 72 , point clouds 10 , and other cues 68 ; 47 ; 71 , while NeRF stylization transfers reference appearances to 3D scenes 64 ; 65 ; 48 ; 26 ; 15 . Instruction-based approaches leverage 2D diffusion editors like InstructPix2Pix 7 , applying SDS-like guidance 39 ; 55 ; 79 ; 31 ; 27 or Iterative Dataset Update 22 ; 63 ; 61 ; 66 ; 13 ; 9 for improved consistency. Some methods 21 ; 17 ; 37 further enforce alignment using depth information from the trained NeRF, while others 12 ; 70 ; 49 operate directly in a learned 3D latent space. While effective for 3D editing, the reliance of these approaches on dense input views or an initial 3D representation makes them less suitable in sparse-view scenarios. Sparse multi-view editing. In the absence of a full 3D representation, several works have explored editing a set of input images directly. A prominent direction adapts pre-trained diffusion-based monocular editors by modifying self-attention layers: as first shown in 69 , extending queries to attend across frames promotes consistency between the outputs. Building on this idea, a number of methods generate edited image sequences 20 ; 56 ; 33 ; 8 ; 51 ; 40 ; 78 . While effective for temporally smooth sequences with small viewpoint changes, these approaches struggle in the sparse-view setting, where edits must remain consistent under significant viewpoint differences. DGE 11 combines extended attention with 3D Gaussian Splatting (3DGS) lifting: attention-based edits provide rough multi-view consistency, while 3DGS is used to consolidate outputs and resolve residual artifacts. However, in the sparse-view regime, 3DGS tends to overfit the limited input views rather than serve as a true cross-view aggregator 73 ; 77 , leading to persistent inconsistencies. As a result, DGE effectively reduces to an extended-attention approach, inheriting the same limitations as prior video editing methods. Most recently, 5 proposed a feed-forward approach that propagates a user-provided 2D edit to

Read the complete text