Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System
Andreas Madsack, Johanna Heininger, Adela Schneider, Christian Eckard, Robert Weißgraeber · 2025
Read Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System by Andreas Madsack, Johanna Heininger, Adela Schneider, Christian Eckard, Robert Weißgraeber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-16135.txt.
Introduction
Madsack and co-authors evaluate a way to make one data-to-text system speak many languages without rebuilding it for each one. Their approach translates the system's grammar settings from the source language into each target language ahead of time. Machine translation does the first pass, a human reviews it once, and a cross-language model of grammar dependencies ties the setup together. At run time, the system then generates text in the target language with no human in the loop.
The test bed is a basketball data set, and the authors also propose a way to grade the translated output through human post-editing. Their system performed well, with strong grammatical correctness in the generated text. For builders of reporting tools that must serve many markets, the paper offers a route to scale: pay the translation cost once, up front, rather than on every document.
Catalog record
| CREATOR | Andreas Madsack, Johanna Heininger, Adela Schneider, Christian Eckard, Robert Weißgraeber |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-16135
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System Andreas Madsack, Johanna Heininger, Adela Schneider, Christian Eckard, Robert Weißgraeber Abstract One approach for multilingual data-to-text generation is to translate grammatical configurations upfront from the source language into each target language. These configurations are then used by a surface realizer and in document planning stages to generate output. In this paper, we describe a rule-based NLG implementation of this approach where the configuration is translated by Neural Machine Translation (NMT) combined with a one-time human review, and introduce a cross-language grammar dependency model to create a multilingual NLG system that generates text from the source data, scaling the generation phase without a human in the loop. Additionally, we introduce a method for human post-editing evaluation on the automatically translated text. Our evaluation on the SportSett:Basketball dataset shows that our NLG system performs well, underlining its grammatical correctness in translation tasks. 1 Introduction Deep learning-based approaches offer promising results in the field of multi-language data-to-text generation. Being prone to hallucinations, they require a manual review post generation. To address this challenge, we propose a hybrid approach that integrates the rule-based system with deep learning models to leverage the strengths of each while avoiding hallucination. Our motivation behind incorporating the rule-based approach is to ensure accurate multilingual output, since guaranteeing accuracy is the key to scaling multi-language data-to-text systems - avoiding the human in the loop for each produced text. The essential component of our approach is the configurable grammar unit, which carries linguistic information and can be transferred across languages. In this paper, our primary goal is to evaluate the effectiveness of transferring these grammar units across languages during translation. Since grammar transfer is the key to effectively adapting our rule-based system Weißgraeber and Madsack 2017 to multi-language settings, it directly influences the accuracy of our generated texts, and therefore, is worth to be evaluated. We aim to answer two research questions: firstly, how can we effectively evaluate our data-to-text system which incorporates NMT with grammar transfer? To answer that, we developed an evaluation framework, where we track edits of human translators after automatic translation. Secondly, how good is our grammar transfer feature in this hybrid system? We answer it by analyzing what types of post-edits are needed after the automatic translation, which helps us to precisely identify the errors in our system, with a specific focus on the grammar transfer mechanism. 2 System description 2.1 Data-to-Text System A data-to-text NLG system receives as input (a) data and (b) a specification how to generate the resulting human language text. Such a system can be used to automate journalistic texts, financial reports or product descriptions in e-commerce. Correctness is essential for all of these kinds of texts. Therefore, controlling the grammar features to ensure a correct grammatical specification of the system is worthwhile. The NLG generation process has three fundamental steps ( Reiter et al. 2000 ) : Document Planning, Microplanning, and Surface Realisation. Document Planning is about which data are actually part of the resulting text and its overall structure. Microplanning is about the grammatical specification needed to generate an error-free human language text in the third step, Surface Realisation. In our system, the Document Planning stage is based on the sentences the user annotates: which data fields are used and which sentences are part of their story. As for grammatical specification in Microplanning, it is done by the user but with the assistance of our NLG platform, which provides only valid options for the target language in the user interface. For example, the user can decide to use the number of goals from the data and construct the following sentence: ”Peter scored [number of goals] [goal + number=number of goals].” Configurability – in the Surface Realisation step, the grammatical specifications are also needed for the variables of the sentence to be in the correct grammatical agreement. Surface Realisation is a completely automatic step based on the given specifications and on extensive language knowledge. Examples of such systems are Gatt and Reiter 2009 for English, Braun et al. 2019 for German, and commercial ( Dale 2020 ) and proprietary ones ( Weißgraeber and Madsack 2017 ) . 2.2 Multilinguality For data-to-text projects that require the generation of text in more than one language, there are various approaches available. One way to achieve text in multiple languages, is by translating the generated text of a source language to a target language, another way is translating the generation rules from source to target language - the latter being more common for rule-based systems. In either case, Neural Machine Translation is the state-of-the-art for translating plain text. That is, a neural network based on a Transformer architecture Vaswani et al. 2017 is used to translate a sequence of input tokens from the source language to a sequence of tokens in the target language in an end-to-end manner. Those networks are trained on a large set of text pairs and the trained model is then applied to each new translation task. 2.3 Grammar Transfer When translating NLG projects on our platform, the grammatical specifications for the variable parts also need to be translated. Technically, we introduce the grammar unit as a container of grammar settings. Grammar units not only contain the grammar setting (e.g., number, case, tense, or possessor) but also can be transferred from one language to another language. To be able to do this, we use Natural Language Processing with the help of spaCy ( Montani et al. 2023 ) on these grammar units. As Figure 1 shows, after the source text was translated by NMT, each previously marked text snippet belonging to a grammar unit is analysed by SpaCy with the corresponding language-specific model and a dependency parse of the phrase is returned. Dependency parsing has been a common way to represent the syntactical structure in a sentence for quite some time ( Carroll and Charniak 1992 ) . We post-process the spaCy parse tree with custom aggregations of noun, pronoun, and verb grammar units. Grammatical settings that are transferred can be seen in Table 1 . Features Noun Pronoun Verb lemma x x x case x x - number x x x tense - - x person - - x gender x x x preposition x x - adjectives x - - numerals x - - conjunctions x - - determiners x - - pronoun type - x - Table 1: Features per part of speech Figure 1: Translation workflow, which involves adapting grammatical units during text translation to ensure proper grammatical alignment between source and target languages. Table 2 shows an example of transferring grammar units from English to German. Here the English noun phrase “on Saturday” with the preposition “on” and the noun lemma “Saturday”, which is in nominative singular, is translated to the German noun phrase “am Samstag”. It can be seen that, besides the literal translation, there are two transfers happening: 1) a definite determiner is added to the phrase and 2) the case changes to dative. language prep det noun case English on - Saturday nominative German an definite Samstag dative Table 2: Transfer of the grammar unit “on Saturday” in English to “am Samstag”