DogFit: Do main- g uided Fi ne- t uning for Efficient Transfer Learning of Diffusion Models
Yara Bahram, Mohammadhadi Shateri, Eric Granger · 2025
Read DogFit: Do main- g uided Fi ne- t uning for Efficient Transfer Learning of Diffusion Models by Yara Bahram, Mohammadhadi Shateri, Eric Granger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05685.txt.
Introduction
Yara Bahram and co-authors present DogFit in a 2025 paper on retraining image generators for small new domains. Fine-tuning a diffusion model on a narrow data set often wrecks its range, and guiding it at sampling time costs double computation on every picture.
DogFit bakes the guidance into training instead. It adds a domain-aware offset to the training loss, teaching the model the guided behavior outright. The guidance strength goes in as a small extra input, so users keep the fidelity-versus-variety dial at run time. The design leans on the plain source model's stronger sense of the overall picture. The result is controllable generation for specialist image sets without the usual sampling-time tax.
Catalog record
| CREATOR | Yara Bahram, Mohammadhadi Shateri, Eric Granger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DogFit: Do main- g uided Fi ne- t uning for Efficient Transfer Learning of Diffusion Models Yara Bahram, Mohammadhadi Shateri, Eric Granger Abstract Transfer learning of diffusion models to smaller target domains is challenging, as naively fine-tuning the model often results in poor generalization. Test-time guidance methods help mitigate this by offering controllable improvements in image fidelity through a trade-off with sample diversity. However, this benefit comes at a high computational cost, typically requiring dual forward passes during sampling. We propose the Do main- g uided Fi ne- t uning (DogFit) method, an effective guidance mechanism for diffusion transfer learning that maintains controllability without incurring additional computational overhead. DogFit injects a domain-aware guidance offset into the training loss, effectively internalizing the guided behavior during the fine-tuning process. The domain-aware design is motivated by our observation that during fine-tuning, the unconditional source model offers a stronger marginal estimate than the target model. To support efficient controllable fidelity–diversity trade-offs at inference, we encode the guidance strength value as an additional model input through a lightweight conditioning mechanism. We further investigate the optimal placement and timing of the guidance offset during training and propose two simple scheduling strategies, i.e., late-start and cut-off , which improve generation quality and training stability. Experiments on DiT and SiT backbones across six diverse target domains show that DogFit can outperform prior guidance methods in transfer learning in terms of FID and FD DINOV2 \text{FD}_{\text{DINOV2}} while requiring up to 2× fewer sampling TFLOPS. Introduction Denoising diffusion probabilistic models (DDPMs) ( Sohl-Dickstein et al. 2015 ; Ho, Jain, and Abbeel 2020 ) have emerged as powerful generative models, achieving state-of-the-art (SOTA) results in image synthesis ( Dhariwal and Nichol 2021 ; Rombach et al. 2022 ) , video generation ( Gupta et al. 2024 ; Ho et al. 2022 ) , and image editing ( Kawar et al. 2023 ) . However, producing high-quality and diverse outputs still requires substantial computational and data resources, particularly with large diffusion backbones and image sizes. In practical applications such as personalized generation with limited data, full model training from scratch is infeasible, making transfer learning a critical tool for target adaptation. A common approach in this setting is to fine-tune a pre-trained diffusion model on the target data. While straightforward, this strategy frequently suffers from overfitting and poor generalization due to the limited size and diversity of the target domain. Figure 1: DogFit achieves competitive average FID and \mathrm{FD}_{\text{DINOV2}} when compared to SOTA guidance mechanisms for diffusion transfer learning, all without increasing the computational complexity (sampling TFLOPS). Classifier-free guidance (CFG) is a standard technique for improving generation quality at inference time by modulating the conditional signal ( Ho and Salimans 2022 ; Karras et al. 2024 ) . This technique is often used in its original form even after target adaptation to steer the model towards better generalization. However, CFG assumes access to well-trained conditional and unconditional models, which is difficult to ensure in transfer scenarios where the target domain is small or lacks labels ( Ho and Salimans 2022 ; Zhong et al. 2025 ) . Recent methods like domain guidance (DoG) address these issues by utilizing the unconditional source model as a stronger marginal noise estimator, offering improved target domain alignment and generalization ( Zhong et al. 2025 ; Phunyaphibarn et al. 2025 ) . Nevertheless, both CFG and DoG introduce significant computational overhead at sampling time due to their reliance on dual forward passes ( Ho and Salimans 2022 ; Zhong et al. 2025 ) . On the other hand, model guidance (MG) mitigates CFG’s runtime cost in in-domain settings by injecting the guidance signal directly into the diffusion training objective ( Chen et al. 2025 ; Tang et al. 2025 ) . However, MG inherits the limitations of CFG in transfer learning and hard-codes the guidance strength value during training, restricting control at inference. These limitations raise a key question: can we design effective guidance mechanisms for diffusion transfer learning without incurring computational overhead and still maintaining controllability over the guidance strength? This paper proposes domain-guided fine-tuning (DogFit), a method that injects a domain-aware guidance offset into the training loss during the fine-tuning process. This allows the diffusion model to directly learn the guided direction, removing the need for additional test-time processes. The domain-aware design is motivated by our observation that during fine-tuning, the source model offers stronger marginal noise estimates than the evolving target model. This design encourages the model to step toward the target domain manifold and avoid out-of-distribution generation. To support efficient control over the guidance strength at inference, DogFit conditions the model on the guidance value and exposes it to a range of such values during training. This allows the model to modulate its generation behavior accordingly, enabling fidelity–diversity trade-offs at inference with negligible sampling overhead. We further investigate the optimal placement and timing of the guidance offset during training. Two cost-effective guidance scheduling mechanisms are proposed for DogFit: (1) a late-start strategy that delays guidance until the model has learned sufficiently stable target representations; and (2) a cut-off scheme that restricts guidance to the later denoising steps, where fine-grained domain-specific details are more prevalent. Results over six datasets with different distribution shifts and supervision levels, using two SOTA diffusion backbones, DiT ( Peebles and Xie 2023 ) and SiT ( Ma et al. 2024 ) , show that DogFit can outperform SOTA guidance methods in terms of FID and \text{FD}_{\text{DINOV2}} while reducing their sampling TFLOPS by up to 2× (see Figure 1 ). Results establish DogFit as a practical and efficient guidance method for diffusion transfer learning. Our contributions are as follows: • We propose DogFit, a training-time guidance mechanism for diffusion transfer learning that enables controllable improvement over target domain generation fidelity without requiring additional processes or double forward passes at test-time. • We show that, during fine-tuning, the source domain model offers stronger marginal estimates than the target model, making it better suited for generating guidance signals. We empirically analyze how the timing and placement of this guidance impact training and propose two scheduling strategies that enhance training stability and generation quality. • Extensive experiments are conducted across diverse target datasets on DiT and SiT diffusion backbones, showing that DogFit can outperform SOTA guidance methods in transfer learning. Related Work Efficient Diffusion Models. The iterative denoising process in diffusion models imposes high sampling-time costs ( Shen et al. 2025 ) . To accelerate generation, prior work reduces the number of sampling steps via improved solvers ( Song, Meng, and Ermon 2020 ; Lu et al. 2022 ) , optimized noise schedules ( Zheng et al. 2023 ) , or distillation-based methods that train few(one)-step student models to mimic large multi-step teachers ( Salimans and Ho 2022 ; Yin et al. 2024 ) . Recent efforts ( Jensen and Sadat 2025 ; Hsiao et al. 2024 ; Zhou et al. 2025 ) distill CFG to eliminate test-time dual passes. Our method, similar to CFG distillation, aims to internalize guidance, but does so without the architectural modifications or post-hoc training stages required by distillation methods. Instead, DogFit simply integrates guidance directly into the fine-tuning objective. (a) Ground Truth (b) CFG and MG Directions