Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards
Qingming Liu, Zhen Liu, Dinghuai Zhang, Kui Jia · 2025
Read Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards by Qingming Liu, Zhen Liu, Dinghuai Zhang, Kui Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15684.txt.
Introduction
Qingming Liu, Zhen Liu, Dinghuai Zhang, and Kui Jia wrote this computer graphics paper, published in two thousand twenty-five, on polishing 3D generators with human taste. Diffusion models can draft 3D assets fast, but the results often miss instructions or look off in texture and shape.
Their framework, Nabla-R2D3, fine-tunes a 3D diffusion model using rewards scored on flat 2D renders of its output. No 3D preference data is needed, which makes the loop cheap. The method builds on a principled scheme that matches the model's score function to reward gradients. Rival fine-tuning either fails to settle or games the reward, a cheat called reward hacking. This method instead raises rewards step by step while keeping the model's prior skills. The authors call it a sample-efficient path to 3D content that follows prompts.
Catalog record
| CREATOR | Qingming Liu, Zhen Liu, Dinghuai Zhang, Kui Jia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards Qingming Liu, Zhen Liu, Dinghuai Zhang, Kui Jia Abstract Generating high-quality and photorealistic 3D assets remains a longstanding challenge in 3D vision and computer graphics. Although state-of-the-art generative models, such as diffusion models, have made significant progress in 3D generation, they often fall short of human-designed content due to limited ability to follow instructions, align with human preferences, or produce realistic textures, geometries, and physical attributes. In this paper, we introduce Nabla-R2D3 , a highly effective and sample-efficient reinforcement learning alignment framework for 3D-native diffusion models using 2D rewards. Built upon the recently proposed Nabla-GFlowNet method, which matches the score function to reward gradients in a principled manner for reward finetuning, our Nabla-R2D3 enables effective adaptation of 3D diffusion models using only 2D reward signals. Extensive experiments show that, unlike vanilla finetuning baselines which either struggle to converge or suffer from reward hacking, Nabla-R2D3 consistently achieves higher rewards and reduced prior forgetting within a few finetuning steps. Project page: nabla-r2d3.github.io Figure 1 : Our Nabla-R2D3 can efficiently and robustly finetune 3D-native diffusion models with differentiable reward models learned from human preferences on appearance, geometry and many other attributes. 1 Introduction Recent advances in 3D generative models have enabled non-experts to produce batches of 3D digital assets at low cost for downstream tasks such as gaming, film, and robotics simulation. However, these assets are rarely production-ready and often require post-processing due to low visual fidelity, suboptimal geometry, ethical biases, and poor instruction following (in text-to-3D setups). These issues largely stem from training datasets that diverge from human preferences and 3D constraints. A common solution is to first obtain a reward model that reflects human preferences and later perform reward finetuning on the generative model—a process commonly known as reinforcement learning from human feedback (RLHF). Originally developed for aligning autoregressive language models, reward finetuning has also been successfully applied to diffusion models, which are among the most popular generative models in the vision domain. It is therefore natural to consider applying similar techniques to 3D diffusion models. However, extending RLHF to 3D diffusion models is non-trivial due to the lack of high-quality 3D reward models. In 2D settings, reward finetuning typically relies on 2D reward models; by analogy, 3D finetuning would require 3D reward models. Yet collecting diverse and high-quality 3D data remains a longstanding challenge, making it difficult to build reward models that reflect human preferences on attributes such as aesthetics, geometry, instruction following, and physical plausibility. Inspired by the success of the "lifting from 2D" approach for 3D generation, where one optimizes a 3D shape such that each view is of high likelihood under a pretrained 2D generative model, we propose to similarly finetune 3D-native diffusion models xiang2024structured ; zhang2024clay ; siddiqui2024meshgpt ; Liu2023MeshDiffusion ; Liu2024gshell ; wang2023prolificdreamer with 2D reward models. Using a 3D-native model, we can sample different camera views and perform the lifting operation in an amortized fashion across training instances, rather than optimizing each object individually. However, sampling from 2D views yields high variance during optimization and can lead to instability and overfitting of 3D-native diffusion models. In light of a state-of-the-art reward finetuning method called Nabla-GFlowNet liu2025nablagfn , which proves to be highly robust, efficient and effective on 2D diffusion models, we propose Nabla-R2D3 (short for R eward from 2 D for D iffusion Alignment in 3 D via Nabla -GFlowNet), which adapts this method to finetune 3D-native diffusion models with 2D rewards. We empirically show that our method produces 3D-native models that are better aligned with human preferences and avoid major artifacts, such as unexpected floaters, commonly produced by prior methods. Furthermore, we demonstrate the effectiveness of different 2D reward models for aligning 3D-native diffusion models for different preferences. We summarize our major contributions in this paper below: • To the best of our knowledge, Nabla-R2D3 is the first method to effectively align 3D-native diffusion models with human preferences using only 2D reward models. • We demonstrate several examples of 2D reward models, including appearance-based and geometry-based ones, for aligning 3D-native generative models on different attributes. • Our extensive experiments show that, compared to the proposed vanilla reward finetuning baselines, our Nabla-R2D3 can effectively, efficiently and robustly finetune 3D-native generative models from 2D reward models with better preference alignment, better text-object alignment and fewer geometric artifacts. 2 Related Work Reward Finetuning of Diffusion Models. The earliest attempt at reward finetuning for diffusion models, named DDPO black2024training , views the denoising process of diffusion models as trajectory sampling in a Markov decision process (MDP), in which each state is a tuple of a noisy image and the corresponding time step; a sampled trajectory starts from a random Gaussian noise at time T and, through the iterative stochastic denoising process, reaches a sample image at time 0 . With this MDP defined, DDPO leverages the classical policy gradient method in reinforcement learning (RL) to finetune diffusion models. As reward models are typically learned with neural nets and thus differentiable, DDPO does not effectively leverage the available first-order information in the reward model. To address this issue, methods like ReFL xu2023imagereward and DRaFT clark2024directly treat each sample trajectory as a deep computational graph and, by assigning reward values to the states, use back-propagation to directly optimize model parameters with respect to reward signals in an end-to-end fashion. While efficient in practice, these methods lack probabilistic grounding—their training objectives are not designed to approximate the reward-weighted distribution—and thus tend to overfit to the reward model. A parallel path is to adopt stochastic optimal control (SOC) that frames the alignment objective as an optimal control problem. The SOC approach in theory may achieve ideal results, but the proposed methods uehara2024fine so far are either ineffective or computationally expensive. Recently, new RL-based diffusion finetuning methods are proposed in the framework of generative flow networks bengio2021foundations ; zhang2022generative ; zhang2022unifying ; ma2024baking ; pan2023stochastic ; zhang2023let ; zhang2023diffusiongf ; zhang2024distributional ; yun2025learning (or GFlowNets in short) that builds generative models on a directed acyclic graph to generate samples according to a reward distribution. The resulting finetuning methods, albeit derived in GFlowNet language, are also deeply rooted in (and in many cases equivalent to) soft Q-learning haarnoja2017reinforcement in reinforcement learning. These new RL methods are constructed from first principles and are shown to be effective and efficient in creating finetuned diffusion models that generate diverse samples. 3D Generation via “Lifting from 2D”. Due to the scarcity of 3D data, several works propose generating 3D shapes with only 2D signals. One early work in this direction is Dream Field jain2022zero , which initializes a 3D shape and optimizes the shape with CLIP scores on images rendered from randomly sampled camera views of the shape. Since the CLIP model is not a generative model, it is hard, if not impossible, to yield detailed appearance and geometry in generated 3D shapes. Following Dream Field, DreamFusion poole2023dreamfusion was proposed to use score distillation sampling (SDS) that replaces CLIP scores with diffusion losses on rendered views. Such an approach is used not only for 3D object synthesis from scratch with 2D models, but also for texture generation on known geometry richardson2023texture , 3D object synthesis with video