CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis
Keunwoo Park, Jihye Chae, Joong Ho Ahn, Jihoon Kweon · 2025
Read CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis by Keunwoo Park, Jihye Chae, Joong Ho Ahn, Jihoon Kweon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10993.txt.
Introduction
Keunwoo Park, Jihye Chae, and their colleagues present CLUE, a method for making varied medical images from text while keeping output stable. Image generators need both range and steadiness, but in fields like medicine there are few image types and little data to learn from. CLUE works with fixed-format prompts and needs no extra data.
Built on Stable Diffusion, CLUE adds an encoder that turns images and prompts into style codes fed to a new attention layer. A loss keeps those codes spread in smooth groups, apart from the prompt. On ear infection images, CLUE sharply cut a standard quality gap score and raised recall by a wide margin.
Catalog record
| CREATOR | Keunwoo Park, Jihye Chae, Joong Ho Ahn, Jihoon Kweon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10993
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis Keunwoo Park, Jihye Chae, Joong Ho Ahn, Jihoon Kweon Abstract Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training. Although these methods have shown promising results in general domains, they face limitations when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts. Performance was assessed on otitis media dataset. CLUE reduced FID to 9.30 (vs. 46.81) and improved recall to 70.29% (vs. 49.60%). A classifier trained on synthetic-only data at 1000% scale achieved an F 1 F_{1} score of 83.21% (vs. 73.83%). Combining synthetic data with equal amounts of real data achieved an F 1 F_{1} score of 94.76%, higher than when using only real data. On an external dataset, synthetic-only training achieved an F 1 F_{1} score of 76.77% (vs. 60.61%) at 1000% scale. The combined approach achieved an F 1 F_{1} score of 85.78%, higher than when using only the internal dataset. These results demonstrate that CLUE enables diverse yet stable image generation from limited datasets and serves as an effective data augmentation method for domain-specific applications. [orcid=0000-0002-1098-9081] [orcid=0000-0001-6726-8894] [orcid=0000-0003-1831-7615] keywords Deep learning ,Text-to-image synthesis ,Medical image generation ,Diffusion ,Style embedding ,Data augmentation † † credit: Conceptualization, Methodology, Software, Visualization, Writing – original draft † † credit: Validation † † credit: Data curation, Supervision, Validation † † credit: Project administration, Supervision, Writing – review and editing † † corresponding: Corresponding author 1 Introduction Generative deep learning has achieved remarkable progress in recent years and has attracted significant interest in various application domains [ 24 ] . Of these advances, text-to-image synthesis—the generation of visually coherent and semantically consistent images from textual descriptions—stands out as both compelling and challenging [ 22 , 3 ] . Recent models such as DALL·E [ 13 , 8 ] , Imagen [ 16 ] and Stable Diffusion [ 14 , 2 ] demonstrate impressive general purpose image generation capabilities. They leverage large scale image-prompt paired datasets to produce visuals that are both diverse and high quality. While these models have shown remarkable performance, adapting them to specialized domains remains challenging. Fine-tuning is often required to align the model with domain-specific data distributions and stylistic conventions. This requirement is particularly pronounced in medical imaging, where applications demand not only high visual fidelity but also robust stability and precision due to clinical decision-making. As illustrated in Figure 1 , the medical domain presents distinct challenges for text-to-image synthesis. First, there exists a significant visual and structural gap between medical and general domain images [ 26 ] . Second, diagnostic reports, although clinically equivalent, are written using heterogeneous phrasing conventions, which introduces semantic variability that can confuse generative models. While fine-tuning a model on reports written in one convention (e.g., Report A) allows accurate generation for similar inputs, the model may fail to generalize when exposed to unseen prompts (e.g., Report B), resulting in unpredictable outputs. These constraints consequently render both the diversity and precise control of generated images particularly challenging when image synthesis is conducted solely through the use of identical prompts. Figure 1: Left: Key characteristics of medical datasets. (1) There are notable visual differences between medical and general domain images. (2) Even for the same diagnosis, clinical reports A, B, and C may use different phrasing or expression styles. (3) A model trained on general domain images is not directly applicable to medical image generation. Right: Uncertainty in generative models. When fine-tuned on report A, the model generates accurate images with A-style descriptions, but may produce inappropriate outputs when encountering an unfamiliar phrasing style such as report B. Instability is rooted in the text-to-image models; their generative behavior is tightly coupled to the prompt distribution observed during training. Much of the domain knowledge is implicitly encoded in the training data and thus remains inaccessible through prompts alone. This issue becomes especially problematic in clinical settings, where precise generation is crucial, yet available training data is scarce. Recent efforts to enhance controllability, such as GLIDE [ 10 ] , IP-Adapter [ 25 ] , ControlNet [ 27 ] and T2I-Adapter [ 9 ] , introduce auxiliary conditioning through inputs like reference images, pose maps or edge maps. While effective in improving generation control, these methods depend on additional data modalities that may not be feasible in low resource or privacy sensitive medical environments. To address these challenges, we introduce CLUE (Controllable Latent space of Unprompted Embeddings), a novel framework designed to achieve both visual diversity and generation stability without requiring extra data. CLUE augments the Stable Diffusion architecture with a Style Encoder that learns visual attributes from paired images and prompts. These style embeddings are injected into a new second attention layer of the U-Net [ 15 ] , enabling the model to disentangle and control implicit features not captured in the prompt. Kullback-Leibler (KL) divergence [ 7 ] constraints are applied to enforce a Gaussian distribution in the latent space, facilitating smooth and consistent variation in generated images. We validate CLUE on an Asan Medical Center (AMC) otitis media dataset comprising images of normal tympanic membrane, otitis media with effusion (OME), and chronic otitis media (COM). Our model achieves a substantial improvement in generation quality, reducing the Fréchet Inception Distance (FID) score from 46.81 to 9.30 and improving recall from 49.60% to 70.29%. Synthetic images generated by CLUE, when used as training data, significantly improve downstream classifier performance on the real data, achieving an F_{1} score of 83.21%, compared to 73.83% with reference synthetic data. Combining 100% scale synthetic data with equal amounts of real data further achieved 94.76%. For evaluate external generalization, we additionally tested our framework on the AI-HUB otoscopic image dataset. Even on this unseen dataset, models trained exclusively on CLUE-generated synthetic data achieved an F_{1} score of 72.37% (vs. 54.26%), demonstrating robust generalization beyond the institution. When combining 100% scale synthetic data with equal amounts of real data, CLUE achieved 85.78% F_{1} on the AI-HUB dataset, surpassing the 83.76% achieved by a model trained with only the AMC dataset. 2 Related Work The proposed CLUE framework integrates the complementary strengths of variational autoencoders [ 6 ] and diffusion models to improve both fidelity and control diversity in the image synthesis. In this section, we present a comprehensive review of the theoretical foundations and key principles underlying these generative modeling paradigms. 2.1 Variational Autoencoders Variational Autoencoders (VAEs) offer a probabilistic framework for learning a compact latent representation z\in\mathbb{R}^{d} of high-dimensional data x . The encoder q_{\varphi}(z|x) maps each