AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation

Eric Ji, Qiran Hu, Wufei Ma, Sarthak Jain, Yingying Li, Minh N. Do, Yaoyao Liu · 2026

Read AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation by Eric Ji, Qiran Hu, Wufei Ma, Sarthak Jain, Yingying Li, Minh N. Do, Yaoyao Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31204.txt.

Introduction

Ji and six colleagues present AC3S, a diffusion-based system for making synthetic images with accurate 3D structure. Diffusion models give realistic pictures, yet keeping pose and layout exact is hard. Older pipelines guide them with edge maps and similar prompts, but strong guidance can create artifacts and hurt realism.

AC3S adds a self-supervised visual prompt modulator that adjusts ControlNet conditioning strength as needed, so structure is enforced without smothering the image. A multi-agent vision language model setup also writes detailed prompts matched to the geometry. The system produces datasets with exact 2D and 3D labels, and tests show clear gains in image quality and in how well the synthetic data helps later vision tasks.

Catalog record

CREATOREric Ji, Qiran Hu, Wufei Ma, Sarthak Jain, Yingying Li, Minh N. Do, Yaoyao Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generation Eric Ji, Qiran Hu, Wufei Ma, Sarthak Jain, Yingying Li, Minh N. Do, Yaoyao Liu Abstract Synthetic data generation has emerged as a powerful tool for improving data scalability in computer vision. Recent diffusion-based pipelines have demonstrated strong photorealism. However, how to enforce precise 3D structure and pose consistency in generated images remains challenging. Existing methods leverage visual prompts such as edge maps to guide diffusion models, but often suffer from over-conditioning artifacts that degrade image realism and limit dataset quality. In this paper, we present a diffusion-based image generation framework that enforces 3D structural alignment while preserving photorealism through adaptive conditioning. Our framework, Adaptive Conditioning for 3D-Aware Synthetic Data Generation (AC3S), introduces a self-supervised visual prompt modulator that dynamically adjusts the strength of ControlNet conditioning, preventing over-conditioning and enabling the diffusion model to retain its generative expressiveness. To further enhance diversity and semantic consistency, we develop a multi-agent vision language model framework that composes detailed and 3D-aware prompts aligned with the underlying geometric structure. Together, these components enable the scalable generation of high-quality synthetic datasets with accurate 2D and 3D annotations. Extensive experiments demonstrate that our method significantly improves image quality and downstream utility. 1 1 1 Project page: https://ac3s.cvmlgroup.web.illinois.edu/ 1 Introduction Image generation has rapidly advanced in recent years due to the introduction of diffusion models [ 16 , 44 , 40 , 5 , 34 , 29 , 10 , 4 ] . Large-scale pre-trained models can synthesize high-fidelity images spanning diverse domains [ 38 , 42 ] . Beyond creative applications, these models have proven to be powerful tools for synthetic data generation, completely transforming the current landscape [ 33 , 14 , 35 ] . Traditional synthetic datasets rely on physics-based rendering engines [ 7 ] , in which users directly control scene composition, lighting, and camera parameters. Building a photo-realistic scene is achievable, but not a trivial task, especially for large-scale datasets. Diffusion models alternatively offer highly accessible photo-realistic capabilities through learning an underlying image data manifold. However, popular pre-trained text-to-image models lack reliable and precise 3D control. Recently, 3D-DST [ 33 ] bridged this gap with their image generation pipeline. Their method renders an image of a 3D model, extracts its edge map, and treats it as a visual prompt to condition a diffusion model through ControlNet [ 56 ] . The objects in their generated images align in pose with the rendered model, allowing them to acquire 3D labels. While accurate poses are achieved, the framework sacrifices photorealism: backgrounds become blurred, object textures are simplified, and scenes become incoherent. Figure 1 : Illustrated are samples generated using our adaptive pipeline. The top row displays the CAD renders used to extract visual prompts and data labels. We combine the visual prompt with our 3D-aware text prompts to generate the synthetic images in the bottom row. Note that the subject in each synthetic image is precisely aligned with its corresponding render. In this paper, we propose a diffusion-based image generation framework that enforces 3D structural alignment while preserving the photorealistic capabilities of pre-trained diffusion models. Our key observation is that visual prompts derived from CAD renders are often sparse and overly restrictive when injected directly into diffusion models. To mitigate this issue, we introduce an adaptive visual prompt modulator that dynamically adjusts the strength of ControlNet conditioning during generation. The modulator predicts a conditioning scale from the feature magnitudes of the diffusion model and ControlNet, enabling the system to automatically balance structural alignment and generative expressiveness. In addition to visual conditioning, generating diverse and semantically consistent prompts remains an important challenge in large-scale synthetic dataset creation. Template-based prompts often lack detail, while captions extracted from real images may conflict with the visual prompts used for geometric control. To address this issue, we design a multi-agent vision language model framework that composes detailed and 3D-aware prompts aligned with the underlying visual structure. Multiple specialized agents collaborate to identify the subject, infer pose information, and construct scene descriptions to generate both positive and negative prompts. This design ensures that textual guidance remains consistent with the visual prompts and encourages the diffusion model to generate realistic, diverse images. Qualitatively, our pipeline’s adaptive conditioning modules yield superior samples compared to the baseline. We validate such observations through a series of experiments analyzing generative quality and downstream utility. Specifically, we observe an improvement in FID score [ 15 ] by 15.95 points over the baseline and provide up to a 7% boost in accuracy when fine-tuning models with our synthetic data. Furthermore, we show that in the absence of real-world data, training strictly on our synthetic data can still produce functional models. Our contributions can be summarized as follows: • Visual prompt modulator for ControlNet. We introduce a simple yet effective self-supervised training strategy to train a visual prompt modulator. By automatically attenuating ControlNet outputs, we can prevent overconditioning on a per-sample basis, leading to higher quality images. • Multi-agent VLM prompt generator. We propose a novel multi-agent framework to synthesize descriptive and diverse text prompts. By integrating 3D context, we can ensure alignment between text and visual prompts. • Full Synthetic Dataset. Our full synthetic dataset consists of one million images evenly distributed among the 1,000 ImageNet [ 9 ] categories. We provide labels for classification, object pose estimation, and additional metadata. 2 Related Work Text-to-Image Diffusion Models. Pre-trained text-to-image diffusion models excel in generating photo-realistic images from natural language [ 38 , 42 , 17 , 43 , 47 , 46 , 1 ] . These models begin with sampling Gaussian noise, before iteratively passing through a denoising U-Net [ 41 ] until reaching the image manifold. Latent diffusion models [ 40 ] further optimize by operating in a compressed latent representation, addressing the high-dimensionality of images. Conditioning on text allows for guidance during generation towards desired results. Text is highly expressive, but it typically cannot provide fine-grained constraints. As a result, standard text-to-image models struggle with accurate geometric control, motivating alternative prompting techniques like ControlNet [ 56 ] . ControlNet. ControlNet [ 56 ] extends the conditioning capabilities of diffusion models with visual prompts. Examples of visual prompts include edge maps, depth maps, and segmentation masks. Their method is inspired by side-tuning [ 54 ] , an incremental learning [ 28 , 25 , 26 , 23 , 32 , 27 , 57 , 24 , 11 , 10 , 60 , 21 , 19 , 20 ] technique that allows networks to adapt to new tasks by training a lightweight "side" network. For diffusion models, a copy of the U-Net encoder [ 41 ] acts as the "side" network. This ControlNet takes the visual prompt as input and learns to produce features to inject back into the decoder through skip connections. The injections are performed by element-wise addition \hat{f}=f+\mathcal{F}(I_{prompt};\theta) (1) where f denotes the intermediate U-Net features, and \mathcal{F}(v;\theta) denotes the ControlNet outputs produced by visual prompt v with parameters \theta . When training a ControlNet, the diffusion model’s weights are frozen, allowing only the parameters \theta to learn. In practice, we observe that features produced by pre-trained ControlNets are too strong and degrade photorealism, hence motivating

Read the complete text