Mention Attention for Pronoun Translation
Gongbo Tang, Christian Hardmeier · 2024
Read Mention Attention for Pronoun Translation by Gongbo Tang, Christian Hardmeier (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14829.txt.
Introduction
Gongbo Tang and Christian Hardmeier improve pronoun translation in a 2024 paper. Pronouns point back to earlier mentions, and languages differ in how they use them, so machine translation often gets them wrong. Mentions, the phrases a pronoun may point to, sit closer to the pronoun than other words do. Their model adds a module that pays extra attention to source mentions during translation.
Two small classifiers teach the model to spot mentions, and their output guides the extra attention. The decoder also uses target-side context. On an English-German benchmark, the model beat a strong baseline on pronoun measures and overall quality. The additions did not hurt general translation.
Catalog record
| CREATOR | Gongbo Tang, Christian Hardmeier |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14829
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mention Attention for Pronoun Translation Gongbo Tang, Christian Hardmeier Abstract Most pronouns are referring expressions, computers need to resolve what do the pronouns refer to, and there are divergences on pronoun usage across languages. Thus, dealing with these divergences and translating pronouns is a challenge in machine translation. Mentions are referring candidates of pronouns and have closer relations with pronouns compared to general tokens. We assume that extracting additional mention features can help pronoun translation. Therefore, we introduce an additional mention attention module in the decoder to pay extra attention to source mentions but not non-mention tokens. Our mention attention module not only extracts features from source mentions, but also considers target-side context which benefits pronoun translation. In addition, we also introduce two mention classifiers to train models to recognize mentions, whose outputs guide the mention attention. We conduct experiments on the WMT17 English–German translation task, and evaluate our models on general translation and pronoun translation, using BLEU, APT, and contrastive evaluation metrics. Our proposed model outperforms the baseline Transformer model in terms of APT and BLEU scores, this confirms our hypothesis that we can improve pronoun translation by paying additional attention to source mentions, and shows that our introduced additional modules do not have negative effect on the general translation quality. 1 Introduction Pronouns have different functions and their use varies across languages. They can pose serious problems for MT systems, which potentially causes many bad translations depending on language pairs and text types ( Hardmeier and Federico 2010 ; Scherrer et al. 2011 ) . Neural machine translation (NMT) has boosted machine translation significantly in recent years Kalchbrenner and Blunsom 2013 ; Sutskever et al. 2014 ; Bahdanau et al. 2015 ; Luong et al. 2015 ; Gehring et al. 2017 ; Vaswani et al. 2017 . Yet NMT still suffers in pronoun translation, especially anaphoric pronouns ( Müller et al. 2018 ; Guillou et al. 2018 ; Hardmeier and Guillou 2018 ) . In this paper, we target to improve pronoun translation. There are two main challenges in pronoun translation, one is identifying what the pronouns refer to in the source language, and the other one is the agreement of pronouns in gender or number in the target language. Sorodoc et al. 2020 have found that neural language models can capture grammatically features but not semantic referential information. NMT models are language models and can generate fluent sentences with few grammatical errors. Thus, dealing with anaphoric pronouns are more challenging, and building connections between pronouns and other mentions is a key for pronoun translation. Yin et al. 2021 have shown that the Encoder-Decoder attention has very low alignment with supporting context for ambiguous translations. We assume that NMT models do not extract enough features from source-side mentions. Thus, we hypothesize that we can improve pronoun translation by paying additional attention to source mentions. Now it arises another question: How to pay additional attention to mentions? Attention mechanism is very flexible and not all the words are paid attention to equally in NMT. It is not wise to change the attention weights over source mentions directly in the conventional Encoder-Decoder attention. One possible solution is that we can introduce another attention module in the decoder and this attention module only pays attention to source mentions which is different from paying attention to all the tokens twice. We apply simple masking constrains to let the mention attention focuses on mentions instead of using sparse attention ( Martins and Astudillo 2016 ) . To let models have the ability to recognize mentions, we also introduce two mention classifiers, one is in the source side and the other one is in the target-side. We jointly optimize the translation loss and the mention classification loss. The mention tags are generated by existing tools during training, and the mention classifiers predict mention tags by themselves during inference. These mention tags can guide the mention attention. In addition, the inputs to our mention attention module are decoder hidden states which are aware of target-side context, this potentially benefits pronoun translation as well, because human translators consider target-side translations as well. We conduct experiments on the WMT 2017 English–German translation task, our proposed method achieves slightly better BLEU scores compared to the baseline, a conventional Transformer NMT model. As BLEU metric ( Papineni 2002 ) can not indicate the ability to translate pronouns, we use the accuracy of pronoun translation (APT) metric ( Miculicich Werlen and Popescu-Belis 2017 ) and contrastive evaluation to evaluate our models on pronoun translation. Our experimental results show that our proposed method achieves higher APT scores, especially for ambiguous pronouns. These results confirms that we can improve pronoun translation via paying additional attention to source mentions. 2 Methodology 2.1 Models Our model is built upon conventional Transformer NMT architecture, which is illustrated in Figure 1 . Compared to the base architecture, we introduce a mention attention layer in the decoder, on the top of the original feedforward neural network (FFNN) layer, to extract additional features from source-side mentions. Following the base architecture, we add another FFNN layer on the top of mention attention module, before the output layer, which can further extract features. We also introduce two mention classifiers in the encoder and the decoder, to let the model learn to distinguish mentions and other tokens. Each classifier is a FFNN, where the inputs are hidden states from the encoder or the decoder, and the outputs are mention or not. During training, we feed tokens with mention tags to the model, and let the model learn to predict mentions and translate jointly. During inference, the model itself predicts mention tags for each token to guide the mention attention to extract source features. Figure 1: Illustration of model architecture with mention attention module and two mention classifiers. Red blocks and arrows are newly introduced modules compared to Transformer NMT models. 2.2 Mention Attention The mention attention module is used for extracting additional features from source mentions. It has the same architecture of Encoder-Decoder attention. However, the mention attention does not extract features from all the source tokens but only mention tokens. We implement it through mention masking, which is affected by mention tags, either directly from input or predicted by the mention classifiers. The input to the mention attention is the output of the first FFNN which has considered both source- and target-side context. This helps our mention attention better extract additional source-side features. 2.3 Evaluation We apply BLEU and APT metrics to evaluate the performance of the Transformer model and our proposed model, for general translation and pronoun translation. In addition, we also conduct contrastive evaluation to test their abilities to recognize correct and incorrect pronoun translations. APT Since ambiguous pronouns are more challenging, we compute APT scores on both general pronouns and ambiguous pronouns. Here we focus on “it” and “they” following Guillou et al. 2016 ; Loáiciga et al. 2017 . Contrastive Evaluation Evaluation Contrastive evaluation tests the sensitivity of NMT models to specific translation errors. In contrastive evaluation sets, human reference translations are paired with one or more contrastive variants, where a specific type of error is introduced automatically. The evaluation procedure then exploits the fact that