A Transformer-based Neural Architecture Search Method Conference: Genetic and Evolutionary Computation Conference Companion; July 15–19, 2023; Lisbon, Portugal Genetic and Evolutionary Computation Conference Companion (GECCO ’23 Companion), July 15–19, 2023, Lisbon, Portugal DOI: 10.1145/3583133.359
Shang Wang, Huanrong Tang, Jianquan Ouyang · 2025
Read A Transformer-based Neural Architecture Search Method Conference: Genetic and Evolutionary Computation Conference Companion; July 15–19, 2023; Lisbon, Portugal Genetic and Evolutionary Computation Conference Companion (GECCO ’23 Companion), July 15–19, 2023, Lisbon, Portugal DOI: 10.1145/3583133.359 by Shang Wang, Huanrong Tang, Jianquan Ouyang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01314.txt.
Introduction
Shang Wang, Huanrong Tang, and Jianquan Ouyang search for better neural network shapes with a transformer twist. Their 2025 paper automates architecture design for translation models. The search varies how attention is computed and how encoder and decoder blocks combine. A genetic algorithm then improves each candidate in a population over rounds.
Two scores guide the hunt: the standard BLEU mark for translation quality, plus perplexity as a helper measure. The structures the search found beat all baseline models in their tests. Adding the helper score found better models than BLEU alone could. The authors present the method as a way to let evolution, steered by two goals, draft network designs that humans might miss.
Catalog record
| CREATOR | Shang Wang, Huanrong Tang, Jianquan Ouyang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01314
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Transformer-based Neural Architecture Search Method Conference: Genetic and Evolutionary Computation Conference Companion; July 15–19, 2023; Lisbon, Portugal Genetic and Evolutionary Computation Conference Companion (GECCO ’23 Companion), July 15–19, 2023, Lisbon, Portugal DOI: 10.1145/3583133.3590735 ISBN: 979-8-4007-0120-7/23/07 CCS: Computing methodologies Genetic algorithms CCS: Computing methodologies Machine translation Shang Wang, Huanrong Tang, Jianquan Ouyang Abstract . This paper presents a neural architecture search method based on Transformer architecture, searching cross multihead attention computation ways for different number of encoder and decoder combinations. In order to search for neural network structures with better translation results, we considered perplexity as an auxiliary evaluation metric for the algorithm in addition to BLEU scores and iteratively improved each individual neural network within the population by a multi-objective genetic algorithm. Experimental results show that the neural network structures searched by the algorithm outperform all the baseline models, and that the introduction of the auxiliary evaluation metric can find better models than considering only the BLEU score as an evaluation metric. 1. Introduction The Transformer ( Vaswani et al. 2017 ) model has been used with great success in the application of neural machine translation. To further enhance its capabilities, this paper introduces the genetic algorithm-based neural architecture search ( Liu et al. 2021 ) (GA-NAS) technique to the Transformer model, which breaks the fixed number and composition of encoders and decoders. To evaluate the translation effectiveness of the neural network, this paper uses two key metrics - the BLEU score ( Papineni et al. 2002 ) and perplexity. BLEU score, which measures the similarity between the predicted output and the reference translation, is used as the primary evaluation index. Perplexity, which measures the model’s ability to predict the next word in a sequence, is used as an auxiliary evaluation index. To solve this problem, a multi-objective genetic algorithm is applied and denoted as MO-Trans in this paper. 2. Proposed method 2.1. Framework of MO-Trans This paper uses MOEA/D ( Zhang and Li 2007 ) as the algorithmic framework because it retains non-dominated individuals on the EP set in successive populations. Algorithm 1 shows the framework of the proposed MO-Trans method. The function g^{te} described in algorithm 1 is the Tchebyshev function defined in reference ( Zhang and Li 2007 ) . Step 2 calculates the distance between any two weight vectors, and get T weight vectors closest to each weight vector. The most time-consuming steps of algorithm 1 are step 3 and step 7 since the neural network corresponding to coding needs to be trained for evaluating each individual in population. Input: stop rule of algorithm; M neural network evaluation metrics; N uniformly distributed weight vectors \boldsymbol{\lambda_{1}},\boldsymbol{\lambda_{2}},...,\boldsymbol{\lambda_{N}} ; The number of neighbors of each weight vector T. Output: set EP. EP\leftarrow\emptyset ; 1 for each i=\{1,2,...,N\} let B_{i}=\{i_{1},i_{2},...,i_{T}\} , where \boldsymbol{\lambda^{i_{1}}},\boldsymbol{\lambda^{i_{2}}},...,\boldsymbol{\lambda^{i_{T}}} are the nearest T vectors to \boldsymbol{\lambda^{i}} ; 2 Initialize N individual transformer architectures according to the genetic coding strategy and train them to obtain m evaluation indicators,let \boldsymbol{FV_{i}}=\boldsymbol{F}(\boldsymbol{x_{i}}) ; 3 Initialize z=\{z_{1},z_{2},...,z_{m}\} ; 4 for i=1 to N do 5 Randomly select two indexes k and l from B_{i} , apply crossover and mutation operators to generate new individual y from x_{k},x_{l} ; 6 Train individual y to obtain m evaluation metrics, for each j=\{1,2,...,m\} , if z_{j}<f_{j}(y) , let z_{j}=f_{j}(y) ; 7 for each j\in B_{i} , if g^{te}(y|\boldsymbol{\lambda^{j}},z)\leq g^{te}(x_{j}|\boldsymbol{\lambda^{j}},z) , let x_{j}=y and \boldsymbol{FV_{j}}=\boldsymbol{F}(\boldsymbol{y}) ; 8 Remove all vectors in EP that are dominated by \boldsymbol{F}(\boldsymbol{y}) , and add \boldsymbol{F}(\boldsymbol{y}) to EP if none of the vectors in EP dominate \boldsymbol{F}(\boldsymbol{y}) ; 9 if the termination condition is not satisfied, back to line 5, else return EP. Algorithm 1 Framework of MO-Trans 2.2. Gene Encoding Strategy Number of encoder/decoder blocks The baseline transformer model consists of six encoders and the same number of decoders, which are represented as an integer in our coding strategy. Details of blocks We have borrowed from Trans-GA ( Feng et al. 2021 ) the way the encoding and decoding blocks are composed, as shown in Figure 1 a and Figure 1 b, there exists 4 candidate blocks for encoder and 3 candidate blocks for decoder, that the M-MHA rectangle denotes the masked multihead attention layer and the C-MHA rectangle denotes the cross multihead attention layer. It can be noted that without a C-MHA layer, the decoder block will not compute the information from the encoder block, so unlike Trans-GA, this paper does not search for decoder blocks without a C-MHA layer. In addition to encoding the layer types, each MHA layer requires an integer to represent the number of heads and each FFN layer requires an integer to represent the dimensions. Figure 1. Searching space of combinations of blocks Cross way The matrix query of decoder blocks and the key,value of encoder blocks are needed to compute multihead attention. In the baseline transformer model ( Vaswani et al. 2017 ) , all decoder blocks are connected to the last encoder block to compute C-MHA. since the bottom encoder block tends to learn more syntax, and the top encoder block tends to learn more semantics, this paper prefers to connect a decoder block to the encoder block located close to it. In order not to waste the results of each encoder block, the last decoder block is connected directly to the last encoder block, but other decoder blocks just have a higher probability to connect to the encoder block which is near it, the nearer the higher. When the number of encoder blocks and decoder blocks is the same, this is a one-to-one relationship, when they are not the same, the situation may be somewhat different. As shown in figure 2 , whether there are more encoder blocks or more decoder blocks, the last encoder block must be connected to the last decoder block. As for the other blocks, when there are more encoder blocks, the dashed line shows that the two blocks with the same distance from the top have the highest probability weight of being connected to each other, with the weight being reduced by half for each additional distance, while when there are more decoder blocks, the situation is similar, with the two blocks with the same distance from the bottom being the most likely to be connected to each other, with all the extra decoder blocks being connected to the last encoder block. Figure 2. Cases in different numbers of encoder blocks and decoder blocks As mentioned above, a transformer architecture in searching space could be encoded as: \{ne,[te,p1,p2]\times ne,nd,[td,p1,p2,p3,ce]\times nd\} , which ne and nd denotes the number of encoder blocks and decoder blocks respectively, integer te and td denotes the type of candidate encoder block ranges [1,4] and candidate decoder block ranges [1,3] respectively, p1,p2,p3 denotes the number of heads in the MHA layer or the dimension in the FFN layer, integer ce only uses in the decoder block which ranges [1,ne] indicates which encoder block to compute cross multihead attention with. If set ne=nd=6 and both the number of heads in the MHA layer and the dimension in the FFN layer have 2 possible values, the size of searching space will reach 2.5\times 10^{19} . 2.3. Genetic operators During population initialisation, all