Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation Chenyang Huang, Fei Huang, Zaixiang Zheng, Osmar R. Zaïane, Hao Zhou, Lili Mou Abstract Multilingual neural machine translation (MNMT) aims at using one single model for multiple translation directions. Recent work applies non-autoregressive Transformers to improve the efficiency of MNMT, but requires expensive knowledge distillation (KD) processes. To this end, we propose an M-DAT approach to non-autoregressive multilingual machine translation. Our system leverages the recent advance of the directed acyclic Transformer (DAT), which does not require KD. We further propose a pivot back-translation (PivotBT) approach to improve the generalization to unseen translation directions. Experiments show that our M-DAT achieves state-of-the-art performance in non-autoregressive MNMT. 1 1 1 Our code and training/evaluation scripts are available at https://github.com/MANGA-UOFA/M-DAT * * footnotetext: Work partially done during an internship at ByteDance. † † footnotetext: Work partially done while working at ByteDance. 1 Introduction Multilingual neural machine translation (MNMT) aims at using a single model for multiple translation directions Firat et al. 2016 . It has attracted the attention of the research community over the years Bapna and Firat 2019 ; Zhang et al. 2021 , and has been widely applied in the industry Johnson et al. 2017 . Most state-of-the-art MNMT models are based on the autoregressive Transformer ( Vaswani et al. 2017 , AT,) . However, the inference of AT is slow, which results in significant latency in real-world applications Gu et al. 2018 . Recent work applies the non-autoegressive Transformer ( Gu et al. 2018 , NAT,) , which generates target tokens in parallel for efficient inference. However, NAT often generates inconsistent sentences (e.g., with repetitive words). Qian et al. 2021 propose the Glancing Transformer (GLAT), which is trained in a curriculum learning fashion. It tackles the weakness of NAT by focusing less on the training samples that lead to inconsistent generalization. To accelerate multilingual non-autoregressive translation, Song et al. 2022 propose a Switch-GLAT method, which is based on the Glancing Transformer, and is equipped with back-translation for data augmentation. To the best of our knowledge, Switch-GLAT is currently the only non-autoregressive system for multilingual translation. However, it suffers from two drawbacks. First, Switch-GLAT requires sequence-level knowledge distillation ( Kim and Rush 2016 , KD,) in every translation direction, which is inconvenient for multilingual tasks. Second, Switch-GLAT is unable to generalize to unseen translation directions (zero-shot translation), which is an essential aspect of multilingual machine translation systems Johnson et al. 2017 ; Chen et al. 2017 ; Gu et al. 2019 . In this work, we propose a multilingual Directed Acyclic Transformer (M-DAT) approach to non-autoregressive multilingual machine translation. Our system leverages the recent directed acyclic Transformer ( Huang et al. 2022b , DAT,) , which does not rely on KD. In addition, we propose a pivot back-translation (PivotBT) approach for the multilingual translation task. Specifically, we back-translate a target sentence to a randomly selected language to obtain an augmented source sentence. The newly generated source sentence and the original target sentence form a synthetic data sample. We observe that if the back-translation direction (e.g., German \rightarrow Romanian) does not exist in the training set (i.e., zero-shot), the augmented source sentence will be of low quality. Therefore, our proposed PivotBT uses an intermediate language for the back translation (e.g., German \rightarrow English \rightarrow Romanian). Our PivotBT is efficient, as the inference of our non-autoregressive model is fast. We evaluated M-DAT in both supervised and zero-shot translation directions. In the supervised setting, our M-DAT achieves 0.4 higher BLEU scores than the previous state-of-the-art Switch-GLAT, while maintaining fast inference. Moreover, our M-DAT does not require KD, and is convenient to be trained on multilingual datasets. In the zero-shot translation settings, our M-DAT is the first NAT model to effectively generalize to unseen translation directions, and even outperforms a strong autoregressive baseline, which is largely attributed to our proposed PivotBT. 2 Related Work The non-autoregressive Transformer ( Gu et al. 2018 , NAT,) predicts all target words in parallel to achieve fast inference, and has been applied to various text generation tasks, such as machine translation Gu and Kong 2021 ; Huang et al. 2022a , summarization Su et al. 2021 ; Liu et al. 2022a ; Liu et al. 2022b , and dialogue generation Zou et al. 2021 ; Qi et al. 2021 . However, the output quality of NAT models tends to be low Stern et al. 2019 ; Ghazvininejad et al. 2019 , and as a remedy, the Glancing Transformerthe Glancing Transformer ( Qian et al. 2021 , GLAT,) applies an adaptive training algorithm that allows the model to progressively learn more difficult data samples. Sequence-level knowledge distillation ( Kim and Rush 2016 , KD,) is commonly used to improve the translation quality of non-autoregressive models. As shown in Zhou et al. 2020 , KD data are less complex compared with the original training set, which is easier for NAT models. However, Ding et al. 2021 find that the KD process tends to miss low-frequency words (e.g., proper nouns), which results in worse translation for NAT models. Therefore, there is a need to remove the KD process for NAT models. The most related work to ours is Switch-GLAT Song et al. 2022 . It combines the Glancing Transformer and knowledge distillation for multilingual machine translation tasks. In addition, Switch-GLAT is equipped with a back-translation technique to augment training data. Our system is based on the directed acyclic Transformer ( Huang et al. 2022b , DAT,) , which expands the generation canvas to allow multiple plausible translation fragments. Then, DAT selects the fragments by predicting linkages, which eventually form an output sentence. In this way, M-DAT is more capable of handling complex data samples, and does not rely on KD. We propose PivotBT to augment the training data to improve the generalization of M-DAT. Our PivotBT is inspired by online back-translation Zhang et al. 2020 , which extends the original back-translation ( Sennrich et al. 2016 , BT,) by randomly picking augmented directions. Different from the previous work, our approach applies pivot machine translation Cheng et al. 2017 to improve the reliability of the back-translation directions that are unseen in the training set. Figure 1: An example of our PivotBT augmenting a German sentence \mathbf{y} to Romanian \hat{\mathbf{x}} , where English is used as the pivot language. The training and back-translation steps are accomplished by DAT Huang et al. 2022b . N_{\text{dec}} is the number of decoding layers. 3 Methodology Multilingual translation handles multiple translation directions with a single model. Suppose a data sample contains a source sentence \mathbf{x}=(\mathrm{x}_{1},\cdots,\mathrm{x}_{T_{\mathrm{x}}}) and a target sentence \mathbf{y}=(\mathrm{y}_{1},\cdots,\mathrm{y}_{T_{\mathrm{y}}}) , where T_{\mathrm{x}} and T_{\mathrm{y}} denote the lengths. In addition, language tags l_{\text{src}} and l_{\text{tgt}} are given to indicate the languages of the source and target sentences, respectively. A multilingual machine translation dataset \mathcal{D} can be represented by \{(\mathbf{x}^{(i)},l_{\text{src}}^{(i)},\mathbf{y}^{(i)},l_{\text{tgt}}^{(i)})\}_{i=1}^{K} , where K is the size of the dataset. Our multilingual Directed Acyclic Transformer (M-DAT) has an encoder–decoder architecture. The encoder of M-DAT takes in an input sentence \mathbf{x}^{(i)} and the target language tag l_{\textrm{tgt}}^{(i)} , whereas the decoder predicts the target-language words independently as the translation. 3.1 Directed Acyclic Transformer To train our system without KD, we adapt the recent directed acyclic Transformer ( Huang et al. 2022b , DAT,) , as it does not use KD but achieves comparable performance to autoregressive models on bilingual machine translation tasks (one direction per model). In general, DAT expands its output canvas to generate multiple plausible translation fragments. Further, DAT predicts links to select the fragments, which form an output sentence. As seen in Figure 1 , DAT predicts extra words and forms the final generation “I feel good”, using the predicted links. Suppose the DAT decoder has S generation steps ( S>T_{\mathrm{y}} ). For each step s within 1\leq s\leq S , DAT makes two predictions: word prediction p_{\text{word}}^{(s)}(\cdot) and link prediction p_{\text{link}}^{(s)}(\cdot) . The word prediction p_{\text{word}}^{(s)}(\cdot) gives the distribution over possible words by mapping DAT’s s th decoder state \bm{h}_{s} to the probability distribution over the vocabulary, given by \displaystyle p_{\text{word}}^{(s)}(\cdot)=\operatorname{softmax}(\mathbf{W}_{\text{word}}\bm{h}_{s}) (1) where \mathbf{W}_{\text{word}} is a learnable matrix. The link prediction p_{\text{link}}^{(s)}(\cdot) computes the distribution over the subsequent steps of the s th step, determining which follow-up step should be linked to the s th step. Specifically, the link prediction leverages the attention mechanism Bahdanau et al. 2015 , which compares the s th step’s hidden state \bm{h}_{s} with the states of subsequent generation steps (from s+1 to S ), given by \displaystyle\begin{aligned} p_{\text{link}}^{(s)}(\cdot)=\operatorname{softmax}([\bm{k}_{s}^{\top}\bm{q}_{{s}+1};\cdots;\bm{k}_{s}^{\top}\bm{q}_{{S}}])\end{aligned} (2) where \bm{k}_{s}=\mathbf{W}_{\mathrm{k}}\bm{h}_{s} and \bm{q}_{s}=\mathbf{W}_{\mathrm{q}}\bm{h}_{s} . \mathbf{W}_{\mathrm{k}} and \mathbf{W}_{\mathrm{q}} are learnable matrices. The operation [;] concatenates scalars into a column vector. Given a reference sequence \mathbf{y} in the training set \mathcal{D} , DAT selects T_{\mathrm{y}} of all S generation steps to generate the words in \mathbf{y} , where the selected steps are connected by predicted links. We denote the indices of the selected steps by \bm{a}=(a_{1},\cdots,a_{T_{\mathrm{y}}}) , where 1=a_{1}<\cdots