Is linguistically-motivated data augmentation worth it?
Ray Groshan*, Michael Ginn*, Alexis Palmer · 2025
Read Is linguistically-motivated data augmentation worth it? by Ray Groshan*, Michael Ginn*, Alexis Palmer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03593.txt.
Introduction
Ray Groshan, Michael Ginn, and Alexis Palmer test whether fake training text built with language smarts beats naive fakes, in a paper from two thousand twenty-five. Two low-resource tongues with unlike word building, Uspanteko and Arapaho, anchor the tests. Jobs are translation and line-by-line glossing.
Many fake-data tricks and mixes were run head to head. Smart fakes helped only when they stayed close to the real data's shape. Stray too far and the edge vanished. The verdict tempers hype for costly crafted data. Builders of rare-language tools get a clear rule: match the real spread first. Readers get a rare fair race between two data schools.
Catalog record
| CREATOR | Ray Groshan*, Michael Ginn*, Alexis Palmer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is linguistically-motivated data augmentation worth it? Ray Groshan*, Michael Ginn*, Alexis Palmer Abstract Data augmentation, a widely-employed technique for addressing data scarcity, involves generating synthetic data examples which are then used to augment available training data. Researchers have seen surprising success from simple methods, such as random perturbations from natural examples, where models seem to benefit even from data with nonsense words, or data that doesn’t conform to the rules of the language. A second line of research produces synthetic data that does in fact follow all linguistic constraints; these methods require some linguistic expertise and are generally more challenging to implement. No previous work has done a systematic, empirical comparison of both linguistically-naive and linguistically-motivated data augmentation strategies, leaving uncertainty about whether the additional time and effort of linguistically-motivated data augmentation work in fact yields better downstream performance. In this work, we conduct a careful and comprehensive comparison of augmentation strategies (both linguistically-naive and linguistically-motivated) for two low-resource languages with different morphological properties, Uspanteko and Arapaho. We evaluate the effectiveness of many different strategies and their combinations across two important sequence-to-sequence tasks for low-resource languages: machine translation and interlinear glossing. We find that linguistically-motivated strategies can have benefits over naive approaches, but only when the new examples they produce are not significantly unlike the training data distribution. \noautomath 1 Introduction Figure 1: Two types of approach to data augmentation. Naive augmentation (left) uses random perturbations to produce new examples which are not necessarily grammatically valid, while linguistically-informed augmentation (right) uses linguistic knowledge to constrain synthetic examples to be grammatically valid. Data augmentation refers to techniques that are used to create additional, artificial examples for training machine learning models in order to increase the total amount of training data. Data augmentation has been well-studied in computer vision, where simple perturbations such as flipping, rotating, or recoloring images are applied on natural data ( Lecun et al. 1998 ) . Similar approaches that depend on random perturbation have been used in NLP for tasks such as morphological inflection ( Silfverberg et al. 2017 ; Bergmanis et al. 2017 ; Anastasopoulos and Neubig 2019 ; Yang et al. 2022 ) , classification ( Wei and Zou 2019 ; Karimi et al. 2021 ) , and machine translation ( Wang et al. 2018 ; Guo et al. 2020 ) . One limitation of such approaches is that they often create new examples which are not linguistically valid. For example, a strategy which randomly inserts words might produce an ungrammatical sentence such as "The dog chases bird the cat," where bird is the inserted word. To address this issue, some researchers leverage linguistic resources to produce examples that are both novel and grammatical ( Zhang et al. 2015 ; Wei and Zou 2019 ; Pratapa et al. 2018 ; Seo et al. 2023 ) . Designing linguistically-motivated strategies ( Figure 1 ) generally requires an expert with knowledge of the target language and an understanding of the principles underlying data augmentation. For many low-resource and endangered languages, such experts are rare, and speakers and scholars have many competing obligations. In this work, we examine whether this expert effort is worthwhile by carefully comparing linguistically-motivated strategies with strategies that can be implemented without a language expert. We study a variety of data augmentation methods across two low-resource languages, Arapaho and Uspanteko. We evaluate on translation (to a high-resource language) in both directions and on the task of interlinear glossing , where the model generates a sequence of morphological glosses corresponding to the input sentence Ginn et al. 2023 . To design linguistically-motivated augmentation strategies, our first author, a trained linguist, extensively studied linguistic reference materials for both Arapaho and Uspanteko. We compare non-linguistic augmentation strategies, such as random word insertion or deletion, with our strategies designed to generate grammatically valid examples. We find that the linguistically-motivated strategies can provide small benefits over the non-linguistic approaches in some cases. However, in cases where the linguistic strategies produce examples which are grammatically valid, but rare or unusual, performance is actually worse for the augmented models. We conclude that while the incorporation of linguistic expert knowledge may be beneficial, it must consider both linguistic grammaticality and the target data distribution. Our specific contributions are the following: • A systematic and comprehensive comparison of various linguistic and non-linguistic data augmentation strategies for low-resource machine translation and interlinear glossing. • Analysis of the effect of combining various augmentation strategies. • Analysis of the interaction between the size of the original training set and the benefits from data augmentation. Our code is available on GitHub 1 1 1 https://github.com/lecs-lab/is-ling-augmentation-worth-it and our results are available on WandB. 2 2 2 https://wandb.ai/augmorph 2 Related Work 2.1 Non-linguistic Augmentation Several methods for augmentation have been proposed that do not rely on linguistic knowledge, instead relying on shallow heuristics or statistical methods to produce novel examples, which may or may not be valid utterances in the language. Backtranslation is a common technique in machine translation, where monolingual data in the target language is translated into the source language ( Sennrich et al. 2016 ) , though the resulting examples may not be completely valid. Wang and Yang 2015 generate novel sentences by replacing words with other words that have similar static embeddings. Likewise, Fadaee et al. 2017 seek to produce valid sentences by substituting words that produce high-probability sentences according to a language model. Andreas 2020 perform a similar procedure with sentence fragments, searching for phrases that appear in similar contexts. Wei and Zou 2019 apply word substitutions, deletions, and insertions, performing perturbations that do not necessarily produce valid sentences. Karimi et al. 2021 use a similar approach but manipulate only punctuation marks. Many additional studies have considered similar heuristics for random perturbation ( Silfverberg et al. 2017 ; Wang et al. 2018 ; Anastasopoulos and Neubig 2019 ; Guo et al. 2020 ; Liu and Hulden 2022 ) . 2.2 Linguistic Augmentation While work in the prior section replaces words or phrases according to statistical patterns, other work proposes the use of linguistic resources to identify valid replacements. This has been done with thesauri ( Zhang et al. 2015 ) , WordNet ( Wei and Zou 2019 ) , and (for code-switched text) bilingual lexicons ( Pratapa et al. 2018 ; Winata et al. 2019 ; Tarunesh et al. 2021 ) . Instead of entire words, some research modifies the linguistic features of selected words in each sentence, such as pronominal gender ( Zhao et al. 2018 ) or verbal inflection ( Li and He 2021 ) . Still other research generates entirely synthetic examples by combining morphemes ( Seo et al. 2023 ) or sampling from formal grammars such as finite-state machines ( Lane and Bird 2020 ) and context-free grammars ( Lucas et al. 2024 ) . 2.3 Our Contributions Our work is novel by providing a careful comparison of similar linguistic and non-linguistic strategies. Additionally, most previous work uses shallow knowledge about the language in the form of dictionaries and thesauri, while we utilize a trained linguist and full reference grammars. The closest prior works are Dai and Adel 2020 ; Kashefi and Hwa 2020 , which compare linguistic and non-linguistic