Vector Quantized-Elites: Unsupervised and Problem-Agnostic Quality-Diversity Optimization Constantinos Tsakonas, Konstantinos Chatzilygeroudis Abstract Quality-Diversity algorithms have transformed optimization by prioritizing the discovery of diverse, high-performing solutions over a single optimal result. However, traditional Quality-Diversity methods, such as MAP-Elites, rely heavily on predefined behavioral descriptors and complete prior knowledge of the task to define the behavioral space grid, limiting their flexibility and applicability. In this work, we introduce Vector Quantized-Elites (VQ-Elites), a novel Quality-Diversity algorithm that autonomously constructs a structured behavioral space grid using unsupervised learning, eliminating the need for prior task-specific knowledge. At the core of VQ-Elites is the integration of Vector Quantized Variational Autoencoders, which enables the dynamic learning of behavioral descriptors and the generation of a structured, rather than unstructured, behavioral space grid — a significant advancement over existing unsupervised Quality-Diversity approaches. This design establishes VQ-Elites as a flexible, robust, and task-agnostic optimization framework. To further enhance the performance of unsupervised Quality-Diversity algorithms, we introduce two key components: behavioral space bounding and cooperation mechanisms, which significantly improve convergence and performance. We validate VQ-Elites on robotic arm pose-reaching and mobile robot space-covering tasks. The results demonstrate its ability to efficiently generate diverse, high-quality solutions, emphasizing its adaptability, scalability, robustness to hyperparameters, and potential to extend Quality-Diversity optimization to complex, previously inaccessible domains. I Introduction Fig. 1 : Overview of VQ-Elites Traditional evolutionary algorithms (EAs) have been a cornerstone of optimization, designed primarily to identify a single optimal solution for a given problem. Although effective for well-defined tasks, this narrow focus on singular objectives limits their ability to address complex challenges, such as those in robotics and autonomous systems. This limitation hinders their adaptability in dynamically changing environments, where a diverse range of high-performing solutions is often required. Consequently, traditional EAs struggle in scenarios that demand versatility rather than a single, best solution [ 1 , 2 ] . Quality-Diversity (QD) algorithms address the limitations of traditional evolutionary algorithms by redefining the optimization objective [ 1 , 2 ] . Rather than seeking a single optimal solution, QD algorithms, such as MAP-Elites [ 3 , 4 ] , aim to uncover a diverse set of high-performing solutions across a broad behavioral landscape. This “illumination” approach generates multiple effective parameter vectors, each associated with a unique behavior. By promoting both diversity and quality, QD algorithms enable more flexible and robust applications, particularly in autonomous systems, where a repertoire of high-quality behaviors is essential for adapting to dynamic challenges. For instance, QD algorithms have demonstrated the ability to generate diverse locomotion behaviors in quadruped robots [ 4 , 5 , 6 ] . Despite these advances, QD algorithms face significant limitations, particularly regarding the reliance on predefined or ”hard-coded” behavioral descriptors (BDs). In MAP-Elites, for example, the behavior space is often discretized using a set of user-defined descriptors to categorize behaviors, which imposes constraints on the algorithm’s adaptability. These hard-coded descriptors assume a priori knowledge about the task or environment and may not fully capture the complexity or diversity inherent in high-dimensional, novel tasks. This constraint limits the algorithm’s flexibility, as predefined descriptors may inadvertently filter out behaviors that fall outside of the original descriptors or misrepresent complex solutions. Moreover, for tasks where the behavior descriptors are hard to design or even unknown, the manual design of BDs becomes impractical and can hinder the QD algorithm’s capacity to scale or generalize to new domains. Early approaches to automatic BD extraction relied on large static datasets to learn lower-dimensional representations using techniques like Principal Component Analysis (PCA) or auto-encoders. For instance, Cully and Demiris [ 7 ] utilized a dataset of handwritten digits to train an auto-encoder, creating a low-dimensional latent space that captured the key features of different digits. This latent space then served as the behavioral descriptor space, enabling QD algorithms to generate a diverse set of solutions spanning the learned feature space. This approach allowed a robotic arm to autonomously learn to draw digits without requiring manual definitions of their key characteristics. The challenges posed by predefined BDs and static datasets encouraged the development of unsupervised learning methods for BD extraction. A notable example is the AURORA algorithm [ 8 , 9 ] , which employs neural network-based representation learning to autonomously identify behavior descriptors directly from the solution space. The main innovation compared to using a static dataset is that AURORA uses the data generated by the QD algorithm to train the auto-encoder in an online fashion. This innovation represented a significant leap, allowing QD algorithms to generalize across a broader range of tasks without relying on hand-crafted descriptors, thus enabling more adaptable and versatile QD frameworks. Nevertheless, AURORA still requires a lot of heuristics and extensive fine-tuning to be performant. In this paper, we introduce Vector Quantized-Elites (VQ-Elites), a novel QD algorithm that uses unsupervised learning to autonomously construct meaningful behavior spaces without prior task or environment knowledge (Fig. 1 ). We leverage a Vector Quantized Variational Autoencoder (VQ-VAE) to make our algorithm able to dynamically learn behavior descriptors, thus enabling autonomous discovery of relevant skills and behaviors. By integrating VQ-VAE into the QD framework, our approach organizes the behavior space into a structured grid by automatically clustering latent representations. This grid-based structure addresses the shortcomings of previous unsupervised BD methods by combining flexibility with the advantages of well-structured archives. Moreover, VQ-Elites enhances the scalability of QD algorithms, facilitating exploration in high-dimensional tasks and environments that traditional QD methods struggle to handle. To demonstrate the effectiveness of VQ-Elites, we evaluate its performance across a variety of tasks that highlight its adaptability and robustness in complex scenarios. These include both structured and unstructured space-covering tasks, as well as high-dimensional pose-reaching tasks for robotic manipulators. In each experiment, we assess VQ-Elites’ ability to generate diverse, high-quality behaviors while maintaining an organized archive. The results underscore its strengths in flexibility and robustness, showcasing its potential to advance QD frameworks for autonomous robotic systems and extend their applicability to more complex problem domains. In short: • We propose VQ-Elites , a novel QD algorithm that integrates unsupervised learning to construct a flexible, task-agnostic behavior space. Using VQ-VAE models, VQ-Elites dynamically learns BDs, enabling the autonomous discovery of relevant behaviors across diverse tasks without prior environment-specific knowledge. • VQ-Elites autonomously creates a structured grid in behavior space by clustering latent representations, addressing the limitations of previous unsupervised BD methods while maintaining the benefits of structured archives. • We introduce two components, Behavioral Space Bounding and Cooperation , which drastically improve the convergence of unsupervised QD algorithms. • We evaluate VQ-Elites across diverse tasks, including space-covering challenges, pose-reaching tasks for robotic manipulators, and scenarios involving high-dimensional data, and constrained systems. • Experimental results show that VQ-Elites efficiently generates diverse, high-quality behaviors while preserving a well-organized archive, highlighting its potential to enhance QD frameworks for autonomous robotic systems and extend their applicability to complex problem domains. II Preliminaries II-A Quality-Diversity Algorithms One of the notable strengths of QD algorithms is their ability to provide multiple high-performing solutions, offering users significant flexibility. This flexibility arises from the wide range of solutions available, allowing users to balance complexity, desired behavior, and practical implementation. To express this more formally, consider an objective function that returns a fitness value, f_{\theta} , and a behavioral descriptor vector, \boldsymbol{b}_{\theta} , which defines the behavior: \displaystyle f_{\theta},\boldsymbol{b}_{\theta}\leftarrow f(\boldsymbol{\theta}) (1) In practice, the behavioral descriptor characterizes the solution, while the fitness value measures its quality. To clarify the concept of a BD and the difference to the fitness, consider a scenario where a robot must traverse various locations within an environment. In this case, the robot’s coordinates can be the BD, while the average speed it reaches the targets the fitness value. Bringing these components together, the optimization algorithm generates multiple solutions, each defined by its BD vector, which describes the behavior or outcome, and its fitness value, which quantifies its effectiveness. But, what is our ultimate goal when using a Quality-Diversity algorithm? Let us assume, without the loss of generality, that we have a behavior space (feature space) \mathcal{B} and a fitness function that we want to maximize. The ultimate goal of QD optimization is to find all parameters, \boldsymbol{\theta} , that maximize the fitness function for every \boldsymbol{b}\in\mathcal{B} : \displaystyle\forall\boldsymbol{b}\in\mathcal{B}\qquad\boldsymbol{\theta}^{*}=\argmax_{\boldsymbol{\theta}}f_{\theta} \displaystyle s.t.\qquad\boldsymbol{b}=\boldsymbol{b_{\theta}} (2) As already mentioned, the result of a QD algorithm is a set of solutions. This set is called either the archive , collection , or map . Furthermore, during the optimization process, it is possible that the algorithm can produce a solution with a similar BD to a solution that already exists in the archive. In that case, and since we cannot keep all possible solutions, the two solutions compete, and finally, the fittest one is preserved. Archives in QD algorithms can take various forms. Perphaps the easiest one involves discretizing the behavior space into a grid, where each cell corresponds to a unique solution. Solutions generated by the algorithm are assigned to the closest cell based on the Euclidean distance between the solution’s BD and the cell’s center. This grid-based method is widely used, particularly in the MAP-Elites algorithm, one of the most popular QD optimization approaches. Other archive variations include more abstract and unstructured designs. For instance, a user may define a distance threshold, allowing a solution to be stored if its distance to all existing BDs exceeds the threshold, or if it outperforms competing BDs within the threshold. Algorithm 1 MAP-Elites algorithm procedure MAP-Elites ( [n_{1},...,n_{d}] ) \mathcal{A}\longleftarrow create_empty_archive( [n_{1},...,n_{d}] ) for i=1\to G do \triangleright Initialization: G random \boldsymbol{\theta} \boldsymbol{\theta}= random_solution() add_to_archive ( \boldsymbol{\theta},\mathcal{A} ) end for for i=1\to I do \triangleright Main loop, I iterations \boldsymbol{\theta}= selection( \mathcal{A} ) \boldsymbol{\theta}^{\prime}= variation( \boldsymbol{\theta} ) add_to_archive ( \boldsymbol{\theta}^{\prime},\mathcal{A} ) end for return \mathcal{A} end procedure procedure add_to_archive ( \boldsymbol{\theta},\mathcal{A} ) (p,\mathbf{b})\longleftarrow evaluate( \boldsymbol{\theta} ) c\longleftarrow get_cell_index( \mathbf{b} ) if \mathcal{A}(c)=null or \mathcal{A}(c).p