Evolve to Inspire: Novelty Search for Diverse Image Generation
Alex Inch, Passawis Chaiyapattanaporn, Yuchen Zhu, Yuan Lu, Ting-Wen Ko, Davide Paglieri · 2025
Read Evolve to Inspire: Novelty Search for Diverse Image Generation by Alex Inch, Passawis Chaiyapattanaporn, Yuchen Zhu, Yuan Lu, Ting-Wen Ko, Davide Paglieri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00686.txt.
Introduction
Alex Inch, Passawis Chaiyapattanaporn, Yuchen Zhu, Yuan Lu, Ting-Wen Ko, and Davide Paglieri want image generators to surprise us on purpose. Text-to-image diffusion models paint beautifully but repeat themselves, a real limit for exploration and ideation work. Prompt tricks that chase beauty scores do not fix sameness. Their system, WANDER, imports novelty search, an evolutionary method that rewards difference itself.
WANDER evolves natural language prompts with a language model as the mutation engine, while CLIP embeddings measure how novel each image is. Special emitters steer the search into distinct regions of prompt space, widening the variety further. With a leading image model generating and a compact model mutating, WANDER beat existing evolutionary prompt optimizers on diversity metrics, and ablation studies confirmed the emitters carry much of the gain.
Catalog record
| CREATOR | Alex Inch, Passawis Chaiyapattanaporn, Yuchen Zhu, Yuan Lu, Ting-Wen Ko, Davide Paglieri |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00686
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evolve to Inspire: Novelty Search for Diverse Image Generation Thanks: Now at the University of Oxford Thanks: Now at Tesco Technology Thanks: Now at Microsoft Research Alex Inch, Passawis Chaiyapattanaporn, Yuchen Zhu, Yuan Lu, Ting-Wen Ko, Davide Paglieri Abstract Text-to-image diffusion models, while proficient at generating high-fidelity images, often suffer from limited output diversity, hindering their application in exploratory and ideation tasks. Existing prompt optimization techniques typically target aesthetic fitness or are ill-suited to the creative visual domain. To address this shortcoming, we introduce Wander , a novelty search-based approach to generating diverse sets of images from a single input prompt. Wander operates directly on natural language prompts, employing a Large Language Model (LLM) for semantic evolution of diverse sets of images, and using CLIP embeddings to quantify novelty. We additionally apply emitters to guide the search into distinct regions of the prompt space, and demonstrate that they boost the diversity of the generated images. Empirical evaluations using FLUX-DEV for generation and GPT-4o-mini for mutation demonstrate that Wander significantly outperforms existing evolutionary prompt optimization baselines in diversity metrics. Ablation studies confirm the efficacy of emitters. \workshoptitle The First Workshop on Generative and Protective AI for Content Creation 1 Introduction Text-to-image diffusion models like Stable Diffusion, FLUX and GLIDE excel at generating visually appealing images from text prompts Rombach et al. 2022 ; Black Forest Labs 2024 ; Nichol et al. 2022 . However, a significant limitation of these models is that it can be difficiult to use them to generate diverse sets of images ( Marwood et al. 2023 ) unless specifically directed by a user actively writing specific, diverse prompts. This lack of diversity hinders their utility in applications like ideation or exploration, where quickly generating novel ideas is crucial. Simply repeating the prompt yields similar results, and manually tweaking prompts can lead to unpredictable changes, making systematic exploration difficult. Large Language Models (LLMs) have shown promise in generating diverse prompts for text-based tasks via mutation ( Bradley et al. 2023 ; Samvelyan et al. 2024 ; Faldor et al. 2025 ) , but their application to image generation has primarily focused on optimizing prompts for certain fitness objective such as aesthetic or Natural Language Processing (NLP) task performance ( Meyerson et al. 2024 ; Hao et al. 2023 ; Wu et al. 2024 ; Brade et al. 2023 ; Chen et al. 2024 ) . This contrasts with open-ended exploration, which prioritizes novelty and exploration instead of convergence to a single ‘best’ image, motivating the need for a dedicated approach to systematically enhance image diversity. Figure 1: Our method generates significantly more diverse images than reusing a prompt multiple times. One prominent diversity-seeking method is Quality Diversity through AI Feedback (QDAIF) ( Bradley et al. 2023 ) . QDAIF uses an LLM to rate text and assign it to cells on a MAP-Elites grid ( Mouret and Clune 2015 ) , evolving it using an LLM to produce a set of diverse texts while maintaining quality. However, adapting these approaches to images is difficult. Our preliminary experiments (Appendix A ) show that using Vision-Language Models (VLMs) within a QDAIF framework fails; VLM fail to consistently identify qualitatively novel images or accurately categorize images within a MAP-Elites grid based on visual characteristics. Instead, we propose a novelty search-based approach designed to generate diverse image sets from a single starting point ( Lehman and Stanley 2011b ) . We quantify image novelty using the cosine distance between CLIP image embeddings ( Radford et al. 2021a ) . As a mutation operator to generate new individuals, we use an LLM. Inspired by ( Fontaine et al. 2020 ) , we additionally introduce emitters to this problem setting. Emitters are specialized mutation strategies that guide evolution into distinct areas of the behavior space. In our case, emitters are prompts that instruct the LLM to mutate a text prompt in specific manners (a full list can be found in appendix C ). Our experimental methodology involved comparing Wander against other methods by running each algorithm 10 times on identical prompts. To assess the impact of our design choices, we performed an ablation study over different emitter strategies, conducting 10 runs for each of the 10 initial prompts. Additionally, we evaluated the long-horizon performance of random versus bandit-driven selection by running both methods for 30 generations. Results show that Wander achieves superior image diversity while maintaining reasonable relevance and token efficiency compared to existing prompt optimization baselines. In all, we demonstrate that: • Novelty search using image CLIP embeddings is capable of generating highly-diverse sets of images given a simple text prompt as a starting point. • Introducing human-designed mutation strategies (emitters) enhances the diversity of generated images. • We introduce Wander , a framework for prompt optimization that leverages an LLM as a mutation engine, where randomly sampling from a predefined set of semantic instructions (emitters) proves to be a powerful mechanism for diverse exploration. 2 Related Work 2.1 Fine-Tuning for Diversity Recent works have investigated fine-tuning language models for more diverse generation, using reinforcement learning or distillation approaches ( Hao et al. 2023 ; Zhang et al. 2024 ; Cideron et al. 2024 ; Miao et al. 2024 ) . However, fine-tuning approaches have significant limitations; they require access to model parameters, can have high up-front costs, and need to be re-run from scratch for different models. For this work, therefore, we focus on more flexible black-box approaches that can be evaluated via APIs. 2.2 Evolutionary Strategies The use of evolutionary strategies (ES) has a long history in black-box optimization, where iterative mutation and selection mechanisms drive exploration over complex, high-dimensional spaces. A critical advancement in the evolution literature was the development of novelty search. Unlike traditional objective-driven optimization, novelty search explicitly rewards behavioral diversity, thereby avoiding premature convergence to local optima. The seminal work by Lehman and Stanley 2011b demonstrated that novelty-guided evolution can outperform objective-based search by encouraging exploration away from deceptive regions of the search space. This was later extended ( Lehman and Stanley 2011a ) to introduce local competition to balance exploration (novelty) with exploitation (performance). 2.3 Discrete Prompt Optimization Another line of work explores discrete prompt optimization. Methods like APE ( Zhou et al. 2022 ) adopt a Monte Carlo sampling strategy to iteratively sample and optimize the prompts. APE focuses on exploration while APO focuses more on exploitation by computing gradient with regard to a given sample. Recent advancements better explore the discrete prompt space by incorporating evolutionary strategies, such as genetic differential algorithms and multi-phase mutation ( Guo et al. 2024 ; Cui et al. 2024 ) . However, existing methods for discrete prompt optimization focus on maximizing a fitness function, such as quality or NLP task performance on a development set, as opposed to our problem setting of diverse image generation. 2.4 Open-Ended Prompt Evolution Prior work on evolutionary prompt optimization with LLMs has laid a strong foundation for evolving high-quality prompts. PromptBreeder and Rainbow Teaming ( Fernando et al. 2023 ; Samvelyan et al. 2024 ) use self-referential evolution, in which an LLM simultaneously mutates a population of task- and mutation-prompts. These methods have been shown to find novel solutions to problems including