Diffusing Differentiable Representations

Yash Savani, Marc Finzi, J. Zico Kolter · 2024

Read Diffusing Differentiable Representations by Yash Savani, Marc Finzi, J. Zico Kolter (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06981.txt.

Introduction

Yash Savani, Marc Finzi, and Zico Kolter wrote a paper from 2024 on sampling objects that are defined by smooth, tweakable parameters with the help of image diffusion models. Such objects, from panoramas to 3D scenes, are stored as settings of a renderer rather than as raw pixels. Sampling good settings used to mean chasing the single best fit. Their method instead draws a spread of settings.

The trick is to run the diffusion model's reverse flow in image space, then pull that motion back into the space of settings and update the settings along it. The pullback carries a hidden constraint, since not every image comes from a valid set of settings. Handling that constraint sharply improved the sameness and detail of the results. Across images, panoramas, and 3D neural scenes, their samples beat past methods on quality and variety. No new training was needed.

Catalog record

CREATORYash Savani, Marc Finzi, J. Zico Kolter
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06981

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diffusing Differentiable Representations Yash Savani, Marc Finzi, J. Zico Kolter Abstract We introduce a novel, training-free method for sampling differentiable representations (diffreps) using pretrained diffusion models. Rather than merely mode-seeking, our method achieves sampling by “pulling back” the dynamics of the reverse-time process—from the image space to the diffrep parameter space—and updating the parameters according to this pulled-back process. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects. Our method yields diffreps with substantially improved quality and diversity for images, panoramas, and 3D NeRFs compared to existing techniques. Our approach is a general-purpose method for sampling diffreps, expanding the scope of problems that diffusion models can tackle. 1 Introduction Diffusion models have emerged as a powerful tool for generative modeling ( Ho et al., 2020 ; Song et al., 2022 ; Song et al., 2021 ; Karras et al., 2022 ; Rombach et al., 2022 ) , and subsequent work has extended these models to generate complex objects—such as 3D assets, tiled images, and more. Although such approaches typically require training or at least fine-tuning of the diffusion models for these new modalities ( Wang et al., 2023 ; Luo et al., 2023 ) , two notable exceptions, Wang et al. (2022) and Poole et al. (2022) , have developed methods for training-free production of 3D objects by directly using an image-based diffusion model. Both methods work by optimizing a differentiable representation (diffrep)—in this case, a Neural Radiance Field (NeRF) ( Mildenhall et al., 2020 ) —to produce rendered views consistent with the output of the image-based diffusion model. Unfortunately, the nature of both these methods is that they optimize the diffrep to produce the “most likely” representation consistent with the images; that is, they perform mode-seeking rather than actually sampling from the diffusion model. This results in overly smoothed outputs that lack detail and do not reflect the underlying distribution of the diffusion model. In this paper, we present a novel method for sampling directly in the diffrep space using pretrained diffusion models. The method is training-free, can handle arbitrary diffreps, and performs true sampling according to the underlying diffusion model rather than merely mode-seeking. The key idea of our approach is to rewrite the reverse diffusion process itself in the diffrep parameter space. This is achieved by “pulling back” the dynamics of the reverse time process—from the image space to the parameter space—and solving a (small) optimization problem, implied by the pulled-back dynamics, over the parameters for each diffusion step. To further encourage solver convergence, we identify constraints that the diffrep induces on the samples and use these constraints to guide the reverse process to generate samples from the high-density regions of the diffusion model while satisfying the constraints along the sampling trajectory. Our experiments use a pretrained image-based diffusion model (Stable Diffusion 1.5 ( Rombach et al., 2022 ) ) to generate samples from various classes of diffreps. For example, by sampling SIREN representations ( Vincent, 2011 ) with wrap-around boundary constraints, we can sample total 360-degree panorama scenes, and by sampling NeRF representations ( Mildenhall et al., 2020 ) , we can generate 3D models of many objects. In both settings—as well as in baseline comparisons on simple SIREN-based (non-panorama) image generation—our approach substantially outperforms previous methods such as Score Jacobian Chaining (SJC) ( Wang et al., 2022 ) . Though the problem setting is considerably more difficult without fine-tuning or retraining, sampling diffreps using pretrained diffusion models with our method substantially improves and extends the state-of-the-art in training-free generation methods. 2 Related work and preliminaries 2.1 Differentiable representations (diffreps) Diffreps are a powerful tool for representing complex scenes or manifolds using parameterized differentiable functions to map coordinates of the scene into a feature space that encodes the properties of the scene at those coordinates. Popular instantiations of diffreps include: • SIRENs ( Sitzmann et al., 2020 ) , which implicitly model an image using an MLP with sinusoidal activations to map 2D (x,y) pixel coordinates into RGB colors. • NeRFs ( Mildenhall et al., 2020 ) , which implicitly model a 3D scene using an MLP that transforms 3D (x,y,z) voxel coordinates with view directions (\phi,\psi) into RGB \sigma values. We can render images from different views of the scene by numerically integrating the NeRF outputs along the unprojected rays from the camera. Faster or alternative diffreps also exist for 3D scenes—such as the InstantNGP ( Müller et al., 2022 ) model or Gaussian splats ( Kerbl et al., 2023 ) —although in this work we focus on the basic NeRF architecture. Many other kinds of visual assets can also be formulated as diffreps. For example, we can use diffreps to model panoramas, spherical images, texture maps for 3D meshes, compositions of multiple images, scenes from a movie, and even the output of kinematic and fluid simulations. Many interesting diffreps can be used to render not just one image from the scene but multiple coupled images or even a distribution over images . Given a diffrep, parameterized by \theta\in\Theta , for a scene — such as a SIREN panorama or a NeRF—we can render an image of the scene from a view \pi\in\Pi using a differentiable render function f(\theta,\pi)=\texttt{image}\in\mathcal{X} . To accommodate the multiview setting in our discussion, we consider the “curried” Haskell form of the render function f:\Theta\to(\Pi\to\mathcal{X}) , where f(\theta):\Pi\to\mathcal{X} is a map itself from a view \pi\in\Pi to f(\theta)(\pi)=f(\theta,\pi)=\texttt{image}\in\mathcal{X} . In the case of NeRFs or SIREN panoramas, the view \pi is a continuous random variable drawn from a distribution that—with abuse of notation—we will also call \Pi . To formalize this, let \mathcal{H}\subseteq\mathcal{X}^{\Pi} be a vector space of functions from \Pi to \mathcal{X} . With this definition, we can write the signature of f as f:\Theta\to\mathcal{H} . Although \mathcal{H} is usually larger than \mathcal{X} (and can be even infinite in some cases), we can simplify our notation by equipping \mathcal{H} with an inner product to make it a Hilbert space. This allows us to use familiar matrix notation with \mathcal{H} and hide the view dependence of f(\theta) in \mathcal{H} . To complete this formalization, we lift the inner product on \mathcal{X} to define an inner product on \mathcal{H} : \forall h,g\in\mathcal{H}:\quad\langle h,g\rangle:=\mathbb{E}_{\pi\sim\Pi}[\langle h(\pi),g(\pi)\rangle]=\mathbb{E}_{\pi\sim\Pi}[h(\pi)^{\top}g(\pi)]. This formulation allows us to handle the multiview nature of NeRFs and SIREN panoramas in a mathematically rigorous way while preserving the convenient notation. The partial application of f returns an entire view-dependent function f(\theta)\in\mathcal{H} . Given a set of images (x_{i})_{i\in[N]} of the scene from different views (\pi_{i})_{i\in[N]} , the standard diffrep task is to find a \theta\in\Theta that minimizes \mathcal{L}(\theta)=\sum_{i\in[N]}\|f(\theta)(\pi_{i})-x_{i}\| . Because the diffreps and f are both differentiable, this can be accomplished using first-order solvers, such as gradient descent or Adam. We identify \mathcal{H} with \mathcal{X} in the following sections for pedagogical convenience. This identification simplifies the presentation, provides a more interpretable and intuitive perspective on our method, and is precise when \Pi is a singleton. Because \mathcal{H} is a Hilbert space, the main points of our arguments still hold when we consider a larger \Pi . We describe the specific changes needed to adapt our

Read the complete text