Testing the Limits of Machine Translation from One Book

Jonathan Shaw Dillon Mee Timothy Khouw Zackary Leech Daniel Wilson · 2025

Read Testing the Limits of Machine Translation from One Book by Jonathan Shaw Dillon Mee Timothy Khouw Zackary Leech Daniel Wilson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06665.txt.

Introduction

Jonathan Shaw, Dillon Mee, Timothy Khouw, Zackary Leech, and Daniel Wilson wrote a 2025 paper testing how far machine translation can go for a language with almost no digital resources. Modern language models can learn to translate a new language from materials placed in their context, such as a grammar book. The team chose Kanuri, spoken by many people but barely present online.

They built two evaluation sets, one of health and humanitarian terms and one of general terms. Then they fed the model different combinations of resources, such as a grammar and a dictionary, and measured the translation quality. The study shows what a single book of language materials can and cannot buy for an underserved language.

Catalog record

CREATORJonathan Shaw Dillon Mee Timothy Khouw Zackary Leech Daniel Wilson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Testing the Limits of Machine Translation from One Book Jonathan Shaw Dillon Mee Timothy Khouw Zackary Leech Daniel Wilson Abstract Current state-of-the-art models demonstrate capacity to leverage in-context learning to translate into previously unseen language contexts. Tanzer et al. 2024 utilize language materials (e.g. a grammar) to improve translation quality for Kalamang using large language models (LLMs). We focus on Kanuri, a language that, despite having substantial speaker population, has minimal digital resources. We design two datasets for evaluation: one focused on health and humanitarian terms, and another containing generalized terminology, investigating how domain-specific tasks impact LLM translation quality. By providing different combinations of language resources (grammar, dictionary, and parallel sentences), we measure LLM translation effectiveness, comparing results to native speaker translations and human linguist performance. We evaluate using both automatic metrics and native speaker assessments of fluency and accuracy. Results demonstrate that parallel sentences remain the most effective data source, outperforming other methods in human evaluations and automatic metrics. While incorporating grammar improves over zero-shot translation, it fails as an effective standalone data source. Human evaluations reveal that LLMs achieve accuracy (meaning) more effectively than fluency (grammaticality). These findings suggest LLM translation evaluation benefits from multidimensional assessment beyond simple accuracy metrics, and that grammar alone, without parallel sentences, does not provide sufficient context for effective domain-specific translation. 1 Introduction Since the introduction of transformers Vaswani et al. 2017 , neural machine translation (NMT) systems have grown in both precision and the number of languages that they support. For example, Meta released the No Language Left Behind model Team et al. 2022 , which can translate between 200 languages, followed by Google’s release of MADLAD-400 Kudugunta et al. 2023 , which can translate between 450 languages using their largest model. However, achieving reasonable accuracy, especially on such a wide range of languages, requires tremendous amounts of data. MADLAD-400 was trained on around 3T tokens. There are more than 7000 languages on Earth, of which the vast majority lack sufficient corpora for training NMTs. On the other hand, many languages that lack corpora for training contain at least a grammar sketch, with 1904 languages having a detailed grammar of at least 300 pages Nordhoff and Hammarström 2011 . In light of this, placing a grammar in the context window of a large language model for translation tasks has great appeal. Tanzer et al. 2024 pioneered this approach and provided a benchmark system and result for the Kalamang language, demonstrating a score of 45.8 ChrF when translating from English into Kalamang using some heuristics to account for model context window size. To gauge how good of a result this was, they also had a linguist do the same task, achieving a score of 57.0 ChrF. Gemini Team et al. 2024 extended this work by introducing the Gemini 1.5 Pro model with a context window of 1 million tokens, thus accommodating the entire grammar in the model context window, achieving better than human scores of 59.1 ChrF. Clearly, there is potential for LLMs to improve access to Machine Translation capabilities in novel ways. However, we find that much of the discussion centers on the quality of the model and the grammar, with little attention given either to the nature of the translation task or to the evaluation of the produced translation. To take the research further, we reproduce the results in the Gemini Team et al. 2024 paper and adapt it to serve a new low-resource language: Kanuri. For this language, we experiment with both domain-specific and generalized domain datasets, while incorporating native Kanuri translation judgments to evaluate both fluency and accuracy. 2 Related Work Model evaluation remains a critical area of research. One assumption is that models have not yet reached the theoretical upper bounds of how well they can utilize text resources such as a grammar for downstream tasks. Sánchez et al. 2024 created a benchmark of linguistic puzzles and tested a number of models on it, with the intention of finding out which models are best at linguistic reasoning, and in hopes of spurring the creation of improved models. Of the models they tested, Claude Opus was nearly twice as good as the nearest competitor, Gemini. Kornilov and Shavrina 2024 developed a benchmark for how well a system can retrieve the requested grammatical knowledge from a corpus. Ginn et al. 2024 created a multilingual glossing model for 1800 languages. However, Aycock et al. 2024 cast doubt on whether the models are able to use grammars effectively, demonstrating that the LLM did not appear to use the declarative grammar for translation, although it proved useful for grammatical tasks such as producing Interlinear Glossed Text . For translation tasks, a system given only parallel sentences and a dictionary performed comparably with a system given a grammar with parallel sentences. To follow this up, Hus and Anastasopoulos 2024 extended this analysis to 16 languages, utilizing GPT-4-Turbo’s longer context window of 128k tokens to prompt with increasing combinations of resources, ranging from just the extracted dictionary, including the parallel sentences, or the whole grammar book with dictionary and parallel sentences. They found that in all but three cases, the model performed better when given only the dictionary and parallel sentences from the grammar, rather than the full grammar book. In many cases, the primary focus is on the data provided to the model in context; however, there is some evidence that increased prompt complexity also provides improvements. Zhang et al. 2024 found that adding glossed sentences to the prompt helped accuracy, so Ramon et al. 2024 used the glossing model from Ginn et al. 2024 to provide the gloss for the source sentence to increase accuracy. Zebaze et al. 2025 found that if the source sentence is split into several smaller sentences, and then the translations of these sentences are recombined, the system performs better. Research in the area of effective translation metrics is well established, predating modern NMT tools. Lommel et al. 2014 demonstrate that such evaluations are best thought of as multidimensional, creating a typology of translation error categories that include accuracy, the degree to which the translation accurately represents the meaning of the source, and linguistic conventions (formerly called fluency), the degree to which the target translation accurately represents the accepted norms for use of the language. In the context of machine-translated evaluations, one underlying principle remains clear: Licht et al. 2022 note that automatic metrics are intended to align with human judgments. This is not a straightforward task as automatic metrics such as ChrF and BLEU rely on objective, quantitative evaluations, while human judgments are both subjective (i.e. two native speakers may not equally agree on the suitability of a single translation) and qualitative. To that end, they designed a new metric, XSTS, which focuses on meaning/accuracy, asserting that it is more objective and less variant than fluency evaluations. 3 Background Kanuri Language Kanuri (ISO 639-3: knc) is a continuum of Saharan dialects spoken by approximately 9.7 million people in Central Africa, primarily in northeast Nigeria, southeastern Niger, western Chad, northern Cameroon, and a diaspora community in Sudan Ethnologue . Its historical role as a regional lingua franca has decreased over time in favor of Hausa McKenna n.d. . The

Read the complete text