Aiding Medical Diagnosis through Image Synthesis and Classification
Kanishk Choudhary · 2025
Read Aiding Medical Diagnosis through Image Synthesis and Classification by Kanishk Choudhary (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00786.txt.
Introduction
Kanishk Choudhary builds a system that creates medical images from text and then checks its own work. Trainee doctors learn by viewing many examples, but real collections can be narrow and hard to reach. The system fine-tunes a standard image generator on PathMNIST, a set of colon tissue images in nine types.
A second model, trained as a classifier on the same data, grades each generated image. It reached near-perfect accuracy at naming real tissue types. Images that fail the check are thrown out and made again until they pass. The best generator version scored moderately overall in 2025 tests, with some tissue types perfect and complex ones lagging.
Catalog record
| CREATOR | Kanishk Choudhary |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00786
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aiding Medical Diagnosis through Image Synthesis and Classification Kanishk Choudhary Abstract Medical professionals, especially those in training, often depend on visual reference materials to support an accurate diagnosis and develop pattern recognition skills. However, existing resources may lack the diversity and accessibility needed for broad and effective clinical learning. This paper presents a system designed to generate realistic medical images from textual descriptions and validate their accuracy through a classification model. A pretrained stable diffusion model was fine-tuned using Low-Rank Adaptation (LoRA) on the PathMNIST dataset, consisting of nine colorectal histopathology tissue types. The generative model was trained multiple times using different training parameter configurations, guided by domain-specific prompts to capture meaningful features. To ensure quality control, a ResNet-18 classification model was trained on the same dataset, achieving 99.76% accuracy in detecting the correct label of a colorectal histopathological medical image. Generated images were then filtered using the trained classifier and an iterative process, where inaccurate outputs were discarded and regenerated until they were correctly classified. The highest performing version of the generative model from experimentation achieved an F1 score of 0.6727, with precision and recall scores of 0.6817 and 0.7111, respectively. Some types of tissue, such as adipose tissue and lymphocytes, reached perfect classification scores, while others proved more challenging due to structural complexity. The self-validating approach created demonstrates a reliable method for synthesizing domain-specific medical images because of high accuracy in both the generation and classification portions of the system, with potential applications in both diagnostic support and clinical education. Future work includes improving prompt-specific accuracy and extending the system to other areas of medical imaging. 1 Introduction Medical imaging is a foundational tool for clinical diagnosis and education. Yet, access to a diverse range of high-quality medical images is often limited. Trainees may rely on only a few reference cases and rarely encounter certain conditions, hindering their learning. This creates a need for systems that can generate accurate, representative medical images on demand to augment training datasets and support diagnostic reasoning [1, 2]. I focus on generating realistic images for human training rather than solely developing classification models because human medical practitioners remain central to clinical decision-making due to reasons such as interpretability, patient trust, regulatory requirements, and liability concerns. Generative models have shown promise in producing synthetic medical images, but the precision required in medicine means that any inaccuracy can be misleading or harmful. My goal is to produce synthetic images with both the realism and the semantic accuracy necessary for clinical use. Generative Adversarial Networks (GANs) have been widely used for medical image synthesis and augmentation [1]. Goodfellow et al.’s GAN framework [2] trains a generator against a discriminator to create realistic images, and numerous studies have applied GANs to expand medical training data [1, 3]. For example, Frid-Adar et al. improved liver lesion classification by adding GAN-generated images to a training set, boosting a classifier’s performance [10]. However, GANs often require large specialized datasets and careful tuning [1], and they commonly suffer from instability and mode collapse on limited data [1]. Diffusion models have emerged as a compelling alternative, achieving high-resolution, semantically rich outputs that rival or surpass GANs [4]. Latent diffusion models (e.g., Stable Diffusion) generate images via iterative denoising in a latent space [5], demonstrating state-of-the-art results on natural image synthesis [4]. Yet, a model like Stable Diffusion, trained on broad datasets (e.g., LAION-5B), performs poorly on specialized medical imagery without fine-tuning [6]. If prompted with medical terms out-of-the-box, it may produce anatomically incorrect results (Figure 1) [6]. This gap has spurred efforts to adapt diffusion models to medical domains [6]. This work introduces a self-validating generative framework to address both realism and accuracy in medical image synthesis. I fine-tuned Stable Diffusion v1.5 on a histopathology dataset (PathMNIST [9]) using LoRA [7] to specialize it for colorectal tissue images. I then trained a ResNet-18 classifier [8] on the same dataset to serve as an automated validator. During generation, each output image is evaluated by the classifier; if the predicted class does not match the input prompt, the image is discarded and the model tries again. This iterative loop continues until a generated image is classified correctly, ensuring that only images consistent with the target class are retained. My two-model pipeline (generator + validator) thus enforces semantic accuracy in real time. The result is a set of synthetic images that are not only visually realistic but also labeled correctly, meeting the high standards required for clinical and educational use. Figure 1: Example of Generated vs. Real Images: Comparison between a real histopathology image (right), a synthetic image generated by my system (middle), and an inaccurate image generated by a baseline Stable Diffusion model (left). These images all aim to represent the same tissue type. 2 Background Combining image generation with automated validation in the medical domain is relatively new. Most prior studies focus on generating realistic images and evaluating them indirectly (e.g., by seeing if synthetic data improves a model’s performance) rather than checking each image’s accuracy. GAN-based methods have dominated early work in medical image synthesis [1]. Frid-Adar et al. pioneered using GANs to generate synthetic liver lesions, which improved a classifier when added to the training set. This demonstrated synthetic data’s potential to bolster limited datasets, but generation quality was evaluated only via downstream task performance rather than by verifying each image. More recently, Xue et al. introduced HistoGAN [11], a conditional GAN for histopathology that includes a selective augmentation step. HistoGAN generates images conditioned on class labels and then uses a classifier to filter out low-confidence outputs, adding only the most realistic samples to the training set. This provides a form of post-hoc quality control and leads to improved histology image classification. However, HistoGAN’s validation occurs after generation—to decide which images to keep for augmentation—rather than intervening during the image creation process. Other approaches evaluate synthetic images via classifier metrics or expert review. For instance, some diffusion-based pipelines gauge realism with pathologist surveys and measure how well models trained on synthetic images perform [3]. These underline the importance of quality in medical image synthesis, yet they stop short of enforcing correctness during generation. My approach tightly integrates generation and validation. Unlike prior GAN or diffusion studies that optimize for visual realism or rely on indirect validation (e.g., improved downstream task performance), my system explicitly checks each output against the target label as it is produced. Indirect validation provides only aggregate feedback and cannot pinpoint specific failed images. In contrast, my self-validation mechanism forces the generator to immediately regenerate or discard any output the classifier deems incorrect. Conceptually, this is similar to classifier-guided generation, but rather than using the classifier’s gradients to steer the generation process (as in some diffusion models [4]), I use its predictions to iteratively accept or reject outputs until the criteria are met in real time. I fine-tuned a state-of-the-art diffusion model on a specialized dataset and paired it with a domain-trained validator, creating an automated feedback loop for semantic accuracy in synthetic images. To my knowledge, this is one of the first systems to actively enforce ground-truth consistency (each generated image’s class matches its prompt) during image generation, rather than treating validation as an afterthought. I