ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning

Hongshu Guo, Zeyuan Ma, Jiacheng Chen, Yining Ma, Zhiguang Cao, Xinglin Zhang, Yue-Jiao Gong · 2024

Read ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning by Hongshu Guo, Zeyuan Ma, Jiacheng Chen, Yining Ma, Zhiguang Cao, Xinglin Zhang, Yue-Jiao Gong (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07507.txt.

Introduction

Hongshu Guo and co-authors teach one agent to tune many evolutionary search methods at once, in a paper from two thousand twenty-four. Prior tuners fit one method and failed to carry to others without retraining. Their frame, ConfigX, mixes search parts like blocks to spawn varied methods in training.

A transformer agent learns one shared tuning policy across many tasks by trial rewards. After large pre-training the agent tuned unseen tasks cold, beating strong rivals. Light fine-tuning let it adapt to fresh tasks and keep skill over time. Readers in optimization get a step toward a single all-purpose tuner for search methods.

Catalog record

CREATORHongshu Guo, Zeyuan Ma, Jiacheng Chen, Yining Ma, Zhiguang Cao, Xinglin Zhang, Yue-Jiao Gong
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07507

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning Hongshu Guo, Zeyuan Ma, Jiacheng Chen, Yining Ma, Zhiguang Cao, Xinglin Zhang, Yue-Jiao Gong Abstract Recent advances in Meta-learning for Black-Box Optimization (MetaBBO) have shown the potential of using neural networks to dynamically configure evolutionary algorithms (EAs), enhancing their performance and adaptability across various BBO instances. However, they are often tailored to a specific EA, which limits their generalizability and necessitates retraining or redesigns for different EAs and optimization problems. To address this limitation, we introduce ConfigX, a new paradigm of the MetaBBO framework that is capable of learning a universal configuration agent (model) for boosting diverse EAs. To achieve so, our ConfigX first leverages a novel modularization system that enables the flexible combination of various optimization sub-modules to generate diverse EAs during training. Additionally, we propose a Transformer-based neural network to meta-learn a universal configuration policy through multitask reinforcement learning across a designed joint optimization task space. Extensive experiments verify that, our ConfigX, after large-scale pre-training, achieves robust zero-shot generalization to unseen tasks and outperforms state-of-the-art baselines. Moreover, ConfigX exhibits strong lifelong learning capabilities, allowing efficient adaptation to new tasks through fine-tuning. Our proposed ConfigX represents a significant step toward an automatic, all-purpose configuration agent for EAs. 1 South China University of Technology, 2 Massachusetts Institute of Technology, 3 Singapore Management University 1 Introduction Over the decades, Evolutionary Algorithms (EAs) such as Genetic Algorithm (GA) ( Holland 1992 ) , Particle Swarm Optimization (PSO) ( Kennedy and Eberhart 1995 ) and Differential Evolution (DE) ( Storn and Price 1997 ) have been extensively researched to tackle challenging Black-Box Optimization (BBO) problems, where neither the mathematical formulation nor additional derivative information is accessible. On par with the development of EAs, one of the most crucial research avenues is the Automatic Configuration (AC) for EAs ( Ansótegui, Sellmann, and Tierney 2009 ; Huang, Li, and Yao 2019 ) . Generally speaking, AC for EAs aims at identifying the optimal configuration c^{*} from the configuration space \mathcal{C} of an evolutionary algorithm A , across a set of BBO problem instances \mathcal{I} : c^{*}=\mathop{\arg\max}\limits_{c\in\mathcal{C}}\mathop{\mathbb{E}}\limits_{p\in\mathcal{I}}\left[Perf(A,c,p)\right] (1) where Perf() denotes the performance of a configuration for the algorithm under a given problem instance. Traditionally, the primary research focus in AC for EAs has centered on human-crafted AC mechanisms. These mechanisms, including algorithm/operator selection ( Fialho 2010 ) and parameter control ( Aleti and Moser 2016 ) , have demonstrated strong performance on well-known BBO benchmarks ( Hansen et al. 2010 ) , as well as in various eye-catching real-world scenarios such as Protein-Docking ( Hwang et al. 2010 ) , AutoML ( Vanschoren et al. 2014 ) , and Prompting Optimization of Large Language Models ( Chen, Dohan, and So 2024 ) . However, a major limitation of manual AC is its heavy reliance on deep expertise. To address a specific problem, one often needs to consult experts with the necessary experience to analyze the problem and then design appropriate AC mechanisms (as depicted in the top of Figure 1 ). This impedes the broader application of EAs in diverse scientific or industrial applications. Recently, a novel paradigm called Meta-learning for Black-Box Optimization (MetaBBO) ( Ma et al. 2023 ) , has emerged in the learning-to-optimize community. MetaBBO aims to reduce the reliance on expert-level knowledge in designing more automated AC mechanisms. As shown in the middle of Figure 1 , in MetaBBO, a neural network is meta-trained as a meta-level policy to maximize the expected performance (Eq. ( 1 )) of a low-level algorithm by dictating suitable configuration for solving each problem instance. By leveraging the data-driven features of deep models and the generalization strengths of meta-learning ( Finn, Abbeel, and Levine 2017 ) across a distribution of optimization problems, these MetaBBO methods ( Ma et al. 2024c ; Li et al. 2024 ; Song et al. 2024 ) have shown superior adaptability compared to traditional human-crafted AC baselines. Figure 1: Conceptual overview of different AC paradigms. Despite these advancements, there remains significant potential to further reduce the expertise burden. Current MetaBBO methods often need custom neural network designs, specific learning objectives, and frequent retraining or even complete redesigns to fit different backbone EAs, overlooking the shared aspects of AC across multiple EAs. This leads to the core research question of this paper: Is it possible to develop a MetaBBO paradigm that can meta-learn an automatic, all-purpose configuration agent for diverse EAs? We outline the detailed research objective below: c^{*}_{k}=\mathop{\arg\max}\limits_{c\in\mathcal{C}_{k}}\mathop{\mathbb{E}}\limits_{p\in\mathcal{I}}\left[Perf(A_{k},c,p)\right],k=1,2,...,K (2) where K is an exceedingly large number, potentially infinite. This objective is far more challenging since it can be regarded as the extension of Eq. ( 1 ). Concretely, it presents two key challenges: 1) Constructing a comprehensive evolutionary algorithm space is crucial for addressing Eq. ( 2 ), from which diverse EAs can be easily sampled for meta-training the MetaBBO; 2) Ensuring the generalization capability of the learned meta-level policy across not only optimization problems but also various EAs is imperative. To address these challenges, we introduce ConfigX, a pioneering MetaBBO framework capable of modularly configuring diverse EAs with a single model across different optimization problems (as shown at the bottom of Figure 1 ). Specifically, to address the first challenge, we present a novel modularization system for EAs, termed Modular-BBO in Section 3.1 . It leverages hierarchical polymorphism to efficiently encapsulate and maintain various algorithmic sub-modules within the EAs, such as mutation or crossover operators. By flexibly combining these sub-modules, Modular-BBO can generate a vast array of distinct EA structures, hence spanning a comprehensive algorithm space \mathcal{A} . To address the second challenge, we combine the problem instance space \mathcal{I} and \mathcal{A} to form a joint optimization task space \mathcal{T}:\mathcal{A}\times\mathcal{I} . We then consider meta-learning a Transformer based meta-level policy over moderate optimization tasks sampled from the joint space \mathcal{T} to maximize the objective in Eq. ( 2 ), see Section 3.2 and 3.3 . For each task T=(A_{m},I_{n}) , the Transformer generates configurations by conditioning on a sequence of state tokens corresponding to the sub-modules in A_{m} . Through large-scale multitask reinforcement learning over the sampled tasks, it yields a universal meta-policy that exhibits robust generalization to unseen algorithm structures and problem instances. We summarize our contributions in this paper in three folds: • We introduce ConfigX, the first MetaBBO framework to learn a pre-trained universal AC agent via multitask reinforcement learning, enabling modular configuration of diverse EAs across various optimization problems. • Technically, we present Modular-BBO as a novel system for EA modularization that simplifies the management of sub-modules and facilitates the sampling of diverse algorithm structures. We then propose a Transformer-based architecture to meta-learn a universal configuration policy over our defined joint optimization task space. • Extensive benchmark results show that the configuration policy pre-trained by ConfigX not only achieves superior zero-shot performance against the state-of-the-art AC software SMAC3, but also exhibits favorable lifelong learning capability via efficient fine-tuning. Figure 2: Left : The hierarchical polymorphism in Modular-BBO. Right : Legal/Illegal algorithm examples in Modular-BBO. 2 Related Works 2.1 Human-crafted AC Human-crafted AC mechanisms enhance the optimization robustness of EAs through two main paradigms: Operator Selection (OS) and Parameter Control (PC). OS is geared towards selecting proper evolutionary

Read the complete text