Self-Vocabularizing Training for Neural Machine Translation

Pin-Jie Lin Ernie Chang Yangyang Shi Vikas Chandra · 2025

Read Self-Vocabularizing Training for Neural Machine Translation by Pin-Jie Lin Ernie Chang Yangyang Shi Vikas Chandra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13837.txt.

Introduction

Pin-Jie Lin, Ernie Chang, Yangyang Shi, and Vikas Chandra examine how machine translation models choose their working vocabulary during training. Standard byte-pair vocabularies are fixed before training begins, using statistics alone. The authors noticed that trained models naturally drift toward a smaller, distinct subset of tokens.

Their method, self-vocabularizing training, repeats that drift on purpose. Each round, the model labels training pairs with its own predictions, and a new vocabulary is drawn from what it actually uses. Retraining on the leaner set improved translation scores by up to about one and a half BLEU points. Deeper models used more distinct tokens yet needed smaller vocabularies overall.

Catalog record

CREATORPin-Jie Lin Ernie Chang Yangyang Shi Vikas Chandra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-13837

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Vocabularizing Training for Neural Machine Translation Pin-Jie Lin Ernie Chang Yangyang Shi Vikas Chandra Abstract Past vocabulary learning techniques identify relevant vocabulary before training, relying on statistical and entropy-based assumptions that largely neglect the role of model training. Empirically, we observe that trained translation models are induced to use a byte-pair encoding (BPE) vocabulary subset distinct from the original BPE vocabulary, leading to performance improvements when retrained with the induced vocabulary. In this paper, we analyze this discrepancy in neural machine translation by examining vocabulary and entropy shifts during self-training—where each iteration generates a labeled dataset by pairing source sentences with the model’s predictions to define a new vocabulary. Building on these insights, we propose self-vocabularizing training , an iterative method that self-selects a smaller, more optimal vocabulary, yielding up to a 1.49 1.49 BLEU improvement. Moreover, we find that deeper model architectures lead to both an increase in unique token usage and a 6–8% reduction in vocabulary size. 1 Introduction Vocabulary construction, also known as vocabularization, is essential for many natural language processing tasks that involve neural networks, including neural machine translation (MT), as highlighted in various studies ( Mikolov et al. 2013 ; Vaswani et al. 2017 ; Gehrmann et al. 2018 ; Zhang et al. 2018 ; Devlin et al. 2019 ) . However, past vocabulary learning techniques rely on corpus statistics such as entropy Xu et al. 2020 or frequency counts Sennrich et al. 2016 , without considering contextual information or the model’s ability to represent it. Figure 1: Illustration of self-vocabularizing training: At each iteration, the original dataset D_{0} is segmented using vocabulary V_{t} to form the training set D_{t} . D_{t} is then used to train model M_{t} , which generates a pseudo dataset D^{\prime} . A new vocabulary set V_{t+1} is derived from D^{\prime} , completing the training loop. This process repeats until no further improvements are observed. Despite the success of vocabularization in improving MT model efficiency Xu et al. 2020 , we observe a discrepancy between the original byte-pair encoding (BPE) vocabulary Gage 1994 ( V_{0} ), derived from the initial training data, and the BPE vocabulary induced from pseudo-labeled data ( V_{1} ) (see Figure 1 ). This discrepancy is surprising, as it suggests that MT models implicitly learn a pseudo-“optimal” vocabulary ( V_{1} ) that is substantially different from the original vocabulary ( V_{0} ) and is also smaller in size. For instance, on the IWSLT14 DE-EN dataset, |V_{1}| is approximately 20% smaller than |V_{0}| . Moreover, MT models retrained with the pseudo vocabulary V_{1} outperform those trained on the original vocabulary set. This suggests a need to re-examine the assumptions underlying vocabulary learning techniques such as byte-pair encoding and the marginal utility of vocabularization Xu et al. 2020 , as existing methods may overlook model-data interactions, leaving key optimization factors unaccounted for. In this paper, we aim to understand this discrepancy in neural machine translation models by analyzing shifts in vocabulary and entropy during self-training. To this end, we conduct experiments on two language tasks, comparing the vocabulary sets learned from the original training data and the pseudo-labeled data. Our results suggest limited overlap between the two vocabularies and that pseudo data induces a more optimal vocabulary, enabling further improvements. Furthermore, our findings indicate that the decoder has a limited impact on this vocabulary shift, whereas encoder-based interactions play a crucial role in entropy reduction. This suggests that future vocabulary induction methods should focus more on the cross-attention module. Finally, our study has implications for defining an optimal vocabulary set in language generation. This preliminary study also introduces a simple yet effective technique: iterative self-training to self-select a more optimal vocabulary set for performance gains. In summary, this paper makes the following contributions: (1) We identify a discrepancy between the optimal and pseudo-labeled vocabulary derived from MT models. (2) We analyze shifts in vocabulary and entropy during self-training. (3) We propose a simple approach to obtain a competitive vocabulary set and introduce a self-vocabularizing training algorithm that improves performance. 2 Iterative Self-Vocabularization Current vocabularization techniques adopt two contrasting perspectives: (1) Focusing on frequency or entropy statistics to avoid the computational cost of trial training Xu et al. 2020 , which often neglects important parameters and interactions in the process. (2) Obtaining a more optimal vocabulary set through training Salesky et al. 2020 , but at a higher computational cost. This work combines both perspectives by adopting an entropy-based vocabularization approach while utilizing self-training. In self-training, a base model M_{t} is trained on the dataset to generate predictions for input sequences, which are then used to update the next iteration of the base model M_{t+1} . This process is repeated iteratively with the supervised loss \mathcal{L} from labeled instances He et al. 2019 , where x and y are the source and target texts, respectively: \mathcal{L}=-\mathbb{E}_{{\mathbf{x}}\sim p({\mathbf{x}})}\mathbb{E}_{{\mathbf{y}}\sim p_{{\mathbf{\theta}}^{*}}({\mathbf{y}}|{\mathbf{x}})}\log p_{{\mathbf{\theta}}}({\mathbf{y}}|{\mathbf{x}}), (1) where p({\mathbf{x}}) is the empirical data distribution approximated with samples from D , and p_{{\mathbf{\theta}}}({\mathbf{y}}|{\mathbf{x}}) is the conditional distribution defined by the model. The parameter {\mathbf{\theta}}^{*} is randomly initialized at every iteration. For each training iteration at t+1 , we relearn the BPE vocabulary using the original source and the pseudo target generated with M_{t} (i.e., D^{\prime}=\{({\mathbf{x}},f_{{\mathbf{\theta}}}({\mathbf{x}}))|{\mathbf{x}}\in U\} ). Then, M_{t+1} is trained on D_{t+1} , segmented with the newly derived vocabulary V_{t+1} . Measuring Vocabulary Shifts Subword-based approaches like byte-pair encoding are widely used and have demonstrated strong empirical performance ( Sennrich et al. 2016 ; Al-Rfou et al. 2019 ; Costa-jussà and Fonollosa 2016 ; Lee et al. 2017 ; Ding et al. 2019 ; Liu et al. 2020 ; Ott et al. 2018 ; Kudo and Richardson 2018 ; Wang et al. 2020 ) . These methods construct vocabulary by selecting high-probability subword units. Following Xu et al. 2020 , we define the vocabulary shift as the negative change in entropy normalized by vocabulary size: \frac{-(\mathcal{H}_{M_{t+1}({\mathbf{x}})}-\mathcal{H}_{M_{t}({\mathbf{x}})})}{|V_{t}|}, (2) where M_{t+1}({\mathbf{x}})\rightarrow V_{t+1} and M_{t}({\mathbf{x}})\rightarrow V_{t} represent vocabularies from two consecutive training iterations, with sizes |V_{t+1}| and |V_{t}| . The ratio |V_{t+1}|/|V_{t}| reflects compression in vocabulary size. Corpus entropy \mathcal{H}_{v} with vocabulary V is defined as the sum of token entropy, normalized by the average token length: \mathcal{H}_{v}=-\frac{1}{l_{v}}\sum_{j\in V}p(j)\log p(j), (3) where p(j) represents the relative frequency of token j in the training corpus, and l_{v} is the average token length (i.e., the number of characters per token). 3 Experimental Settings Figure 2: Entropy and performance across self-vocabularizing training iterations. ( Left ) BLEU score ( blue ) consistently improves across iterations. Meanwhile, the self-learned vocabulary reduces corpus entropy ( teal ), indicating a better estimation of token distribution. ( Right ) Vocabulary shift measured by vocabulary overlap ( orange ) between consecutive vocabularies V_{t} and V_{t-1} , showing that the model initially selects a broad set of subwords before consolidating onto a subset of V_{t-1} 2 2 2 Vocabulary overlaps at firs iteration leverages the identical vocabulary where V_{0}=V_{1} . . The type-token ratio (TTR) ( purple ) reflects the diversity of learned semantic units, reported on the training corpus scaled by 1000 . For our experiments, we used the IWSLT14 German-English parallel corpus for both German-to-English (DE-EN) and English-to-German (EN-DE) translation tasks. We preprocessed the data using

Read the complete text