PhytoSynth: Leveraging Multi-modal Generative Models for Crop Disease Data Generation with Novel Benchmarking and Prompt Engineering Approach

Nitin Rai, Arnold Walter Schumann, Nathan Boyd · 2025

Read PhytoSynth: Leveraging Multi-modal Generative Models for Crop Disease Data Generation with Novel Benchmarking and Prompt Engineering Approach by Nitin Rai, Arnold Walter Schumann, Nathan Boyd (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01823.txt.

Introduction

Nitin Rai and co-authors generate synthetic crop disease images with text-to-image models and benchmark the computing cost, in a paper from 2025. Field photos of sick crops are slow and costly to gather, yet training data must be large. Past synthetic work leaned on image-to-image methods and never priced the hardware burden.

The team trained three versions of a leading diffusion model family with two fine-tuning styles. The medium version won: it made five hundred images from just thirty-six field samples in about ninety minutes. Its memory, power, and energy per image were all measured, a first for this field. The authors recommend that model as the efficient choice for farm image generation.

Catalog record

CREATORNitin Rai, Arnold Walter Schumann, Nathan Boyd
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01823

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PhytoSynth: Leveraging Multi-modal Generative Models for Crop Disease Data Generation with Novel Benchmarking and Prompt Engineering Approach Nitin Rai, Arnold Walter Schumann, Nathan Boyd Abstract Collecting large-scale crop disease images in the field is labor-intensive and time-consuming. Generative models (GMs) offer an alternative by creating synthetic samples that resemble real-world images. However, existing research primarily relies on Generative Adversarial Networks (GANs)-based image-to-image translation and lack a comprehensive analysis of computational requirements in agriculture. Therefore, this research explores a multi-modal text-to-image approach for generating synthetic crop disease images and is the first to provide computational benchmarking in this context. We trained three Stable Diffusion (SD) variants—SDXL, SD3.5M (medium), and SD3.5L (large)-and fine-tuned them using Dreambooth and Low-Rank Adaptation (LoRA) fine-tuning techniques to enhance generalization. SD3.5M outperformed the others, with an average memory usage of 18 GB, power consumption of 180 W, and total energy use of 1.02 kWh/500 images ( ≈ \approx 0.002 kWh/image) during inference task. Our results demonstrate SD3.5M’s ability to generate 500 synthetic images from just 36 in-field samples in ≈ \approx 1.5 hours. We recommend SD3.5M for efficient crop disease data generation. 1 Introduction Present agricultural robotic technologies rely heavily on computer vision algorithms for smart decision-making. However, to develop reliable and adaptable computer vision algorithms, a diverse set of very large-scale datasets are needed. The current practice of acquiring large-scale datasets in agriculture is primarily based on making multiple field trips during the growing season, which can be a time-consuming and labor-intensive procedure [ 8 , 16 ] . Moreover, even if the datasets are made open access, addressing the specific need of the user–such as dynamic environmental conditions and multiple crop growth stages, is a bottleneck. Thus, alternative methods for automating data generation from small sample collections are necessary. Generative artificial intelligence (Gen AI) offers a promising approach for creating large-scale datasets from a very few samples. As a subfield of AI, it can mimic real-world images to generate synthetic data by capturing fine-grained features representative of specific classes. However, the use of multi-modal generative models (GMs) for image generation remains underexplored, particularly in dynamic crop disease context. Therefore, in this research, we employ a multi-modal GM to generate synthetic images of common diseases in watermelon crops. Unlike weed datasets [ 3 , 19 , 23 ] , crop disease datasets are more complex due to their variability in symptoms and spatial-temporal diversity, making it challenging for GMs to accurately replicate disease patterns. Most disease-centered images in agriculture are trained using Generative Adversarial Networks (GANs) with image-to-image translation [ 17 , 11 , 16 ] . Furthermore, no study has comprehensively addressed critical aspects of training a GM for disease-related applications, such as memory consumption, power usage, energy dissipation, and time required to generate a desired number of samples . These factors are crucial in agricultural contexts, as understanding can help answer key questions: (a) What strategies should be followed to generate realistic samples from disease datasets? (b) How does prompt engineering play a vital role in generating highly realistic disease images? (c) How can adopting a cost-effective approach in training a GM help small organizations with limited access to high-performance computing? (d) How can flexibility and speed in the generation of synthetic disease samples ensure that AI-driven technologies are capable of addressing pressing on-farm disease threats? (e) How can the creation of a standardized benchmarking framework help users and researchers make informed decisions when selecting a GM for synthetic disease generation? Considering the aforementioned gaps in multiple aspects, this research study aims to address these questions by leveraging Stable Diffusion (SD) models based on text-to-image generation for crop disease image creation. In addition, we are the first research to also report novel benchmarking for computational requirements and prompt engineering strategies needed to generate high quality and relevant crop disease dataset . Overall, Sec. 2 discusses related work and the need for SD models in agriculture, Sec. 3 highlights dataset used and strategies to create a good training set, Sec. 4 explores the inner workings of SD models when trained on crop disease dataset along with Dreambooth and Low-Rank Adaptation (LoRA) finetuning approaches, Sec. 5 reports relevant metrics, and finally, Sec. 6 & 7 present findings and observations, and conclusions, respectively. The key contributions are: 1. Train and fine-tune multiple variants of the multimodal SD models using Dreambooth and LoRA techniques to generate realistic synthetic crop disease samples. 2. Measure and report the computational resources (e.g., GPU memory, power consumption, and energy) required to train and generate synthetic crop disease images. 3. Establish benchmarking and standardization to determine the best model for generating crop disease images. 2 Related Work 2.1 Generative adversarial network (GAN) models With advances in synthetic data generation, GAN-based models have proven to be beneficial for agriculture. Many studies have explored various GAN variants with Cycle-Consistent (CycleGAN) and Deep Convolutional (DCGAN) being the most widely used for generating synthetic samples. For example, Yunong et al. [ 35 ] trained CycleGAN to perform data augmentation by generating diseased samples of apple fruit. Similarly, Asenovic et al. [ 1 ] trained a DCGAN to generate synthetic data of multiple crop diseases. Another research by Wu et al. [ 36 ] demonstrated the applicability of generating tomato leaf diseases using DCGAN models. In general, a comprehensive review study by Lu et al. [ 16 ] examined the application of GAN architectures for disease data generation, showcasing the use of more than 15 different variants to generate synthetic samples of various crop diseases. However, GAN architectures have been found to pose several challenges in areas such as training instability [ 7 ] , generating low-quality images [ 42 ] , and demand for large volumes of training set [ 32 ] . Based on the study by [ 16 ] , it is clearly reported that although GAN is the most adopted approach to generate synergetic image data, it has limitations in various areas. Therefore, this research does not rely solely on using GANs but uses state-of-the-art (SoTA) diffusion models for text-to-image generation. 2.2 Diffusion models With the growing interest in using diffusion models to generate high-quality images in the medical field [ 12 ] , product generation [ 14 ] , and urban applications [ 27 ] , application in the agricultural domain remains underexplored [ 21 ] . Although recent studies have explored the potential to leverage diffusion architectures to generate high-quality synthetic samples of crop diseases, no research has demonstrated its comprehensive benchmarking in generating synthetic samples. For example, a study by Mori et al. [ 20 ] proposed using a latent diffusion model to generate images of diseased apple leaves employing an image-to-image generation approach. Another study by Zhou et al. [ 41 ] addressed data scarcity by integrating the diffusion model with a few-shot learning technique to implement an end-to-end pipeline for plant disease identification. A regression-conditional based image-to-image diffusion model was developed by Egusquiza et al. [ 4 ] to create graded synthetic data to quantify plant disease severity. Based on the limited studies that have explored the potential of multimodal GM to generate synthetic disease images, the work of Muhammad et al. [ 21 ] is the only work

Read the complete text