Get away with less: Need of source side data curation to build parallel corpus for low resource Machine Translation
Saumitra Yadav, Manish Shrivastava · 2026
Read Get away with less: Need of source side data curation to build parallel corpus for low resource Machine Translation by Saumitra Yadav, Manish Shrivastava (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08629.txt.
Introduction
Saumitra Yadav and Manish Shrivastava argue for source-side data curation in low-resource machine translation. Their 2026 arXiv paper calls curation a critical yet under-researched step in training translation systems. Training data comes mainly from human translations and digital parallel sources, with limited synthetic generation. For low-resource languages, human translation at scale is prohibitively expensive. Curating the source side well could yield usable parallel corpora with less data. Their title puts it plainly: get away with less by choosing source text with care.
Catalog record
| CREATOR | Saumitra Yadav, Manish Shrivastava |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08629
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Get away with less: Need of source side data curation to build parallel corpus for low resource Machine Translation Saumitra Yadav, Manish Shrivastava Abstract Data curation is a critical yet under-researched step in the machine translation training paradigm. To train translation systems, data acquisition relies primarily on human translations and digital parallel sources or, to a limited degree, synthetic generation. But, for low-resource languages, human translation to generate sufficient data is prohibitively expensive. Therefore, it is crucial to develop a framework that screens source sentences to form efficient parallel text, ensuring optimal MT system performance in low-resource environments. We approach this by evaluating English-Hindi bi-text to determine effective sentence selection strategies for optimal MT system training. Our extensively tested framework, ( L exical A nd L inguistically I nformed T ext A nalysis) LALITA, targets source sentence selection using lexical and linguistic features to curate parallel corpora. We find that by training mostly on complex sentences from both existing and synthetic datasets, our method significantly improves translation quality. We test this by simulating low-resource data availabilty with curated datasets of 50K to 800K English sentences and report improved performances on all data sizes. LALITA demonstrates remarkable efficiency, reducing data needs by more than half across multiple languages (Hindi, Odia, Nepali, Norwegian Nynorsk, and German). This approach not only reduces MT systems training cost by reducing training data requirement, but also showcases LALITA’s utility in data augmentation. keywords Low Resource, Machine Translation, Data curation, Linguistic Feature based sentence selection 1 Introduction The global landscape increasingly demands effective Machine Translation (MT) systems to bridge linguistic divides and facilitate information exchange. Traditionally, the development of robust MT systems has relied heavily on the availability (through human efforts) of vast quantities of parallel data. However, this reliance presents a significant challenge: while major languages benefit from abundant digital resources Tiedemann 2012 ; Kunchukuttan et al. 2018 ; Parida et al. 2020 ; Siripragada et al. 2020 ; Haddow and Kirefu 2020 ; Nakazawa et al. 2021 ; Ramesh et al. 2022 ; Mujadia and Sharma 2022 ; Gala et al. 2023 ; Pal et al. 2023 , numerous other languages, particularly those classified as low-resource and coming from the global south, suffer from a severe scarcity of digitally available data Ranathunga et al. 2023 . A thorough audit of existing multilingual datasets confirms this, revealing that many corpora for low-resource languages are unusable due to a high rate of noise and misalignment Kreutzer et al. 2022 ; Ranathunga et al. 2024 . This disparity creates a fundamental bottleneck, as acquiring high-quality parallel data through human translation for these languages is prohibitively expensive and resource-intensive Jha 2012 , especially for the global south. This situation highlights a critical problem: the very languages that could benefit most from MT are precisely those where data acquisition is most constrained by economic and practical limitations. To address this scarcity, foundational efforts have focused on creating and managing large-scale corpora from the ground up. For many low-resource languages, like Mizo (one of the languages spoken in Mizoram state of India), the primary challenge is the sheer lack of any parallel data. Initial efforts, such as the recent work constructing a large-scale Mizo-English parallel corpus Haulai and Hussain 2023 , focus on creating these foundational resources from scratch. Similarly, the Indian government’s TDIL program and the Indian Language Corpora Initiative (ILCI) Jha 2012 has focused on building a foundational, annotated corpus for key languages. Similarly, Mujadia and Sharma 2022 meticulously constructed Hindi-Telugu parallel Corpus using a multi-pronged approach that included human review and back-translation to ensure quality. These foundational efforts, while monumental, are often a first step, as they do not explore whether a smaller, more strategically selected subset of a corpus could achieve comparable or even superior performance. Our work builds on this progress by addressing the next logical challenge: how to most effectively guide such corpus creation by strategically selecting source sentences for cost-efficient model training. Given the high cost and limited capacity for human translation, it becomes imperative to develop strategies that ensure every translated sentence contributes optimally to the performance of an MT system. The conventional approach of simply gathering more data, irrespective of its content, proves inefficient. This is further highlighted by the availability of large, web-mined corpora. For instance, the Suárez et al. 2019 created the massive OSCAR corpus, providing billions of sentences in 166 languages. Similarly, the creation of the Samanantar corpus Ramesh et al. 2022 prioritized scale by using advanced mining and alignment techniques Feng et al. 2022 . However, this approach, while valuable, may include a large number of simple or redundant sentences that do not meaningfully contribute to a model’s learning. A detailed look at web-mined corpora confirms that their quality is not uniform throughout, with only the highest-ranked portions proving effective for training an NMT model Ranathunga et al. 2024 . This observation challenges the conventional wisdom of simply increasing data volume or aiming for broad diversity, advocating instead for strategic data selection based on inherent linguistic properties. Instead of reactive filtering from noisy parallel data, research must focus on a more discerning selection process: identifying and prioritizing source sentences that yield the greatest learning signal for the MT model. This proactive approach can be seen in the development of tools like OpusFilter Aulamo et al. 2020 , a configurable toolbox for filtering noisy parallel corpora. While effective for general cleaning, such toolboxes often rely on a combination of basic, low-level features that do not fully capture a sentence’s intrinsic linguistic value. Our research addresses this challenge of selecting the most impactful source sentences when human translation resources are fixed, aiming to curate training data that maximizes MT system performance under resource constraints. This strategic shift from merely accumulating data to intelligently curating it holds the potential to democratize MT development, making it feasible for communities with limited resources to build and deploy effective translation systems. This paper introduces LALITA ( L exical A nd L inguistically I nformed T ext A nalysis), a rigorously tested framework designed for source sentence selection in parallel corpus creation. A key innovation within LALITA is the derivation of the LALITA score , a quantifiable metric for sentence complexity 1 1 1 When we refer to a sentence’s complexity, we are primarily discussing its structural and syntactic attributes, such as dependency relations and part-of-speech distributions, rather than its semantic difficulty or its topic. . A higher LALITA score signifies a more structurally complex sentence. The core finding of this work is that by strategically focusing on and curating complex sentences, whether from existing datasets or synthetic sources, LALITA significantly enhances translation quality while drastically reducing the required data volume. The contributions of this research are multi-faceted: • Provides an extensively validated pipeline for source sentence analysis. • Offers a linguistically informed method for augmenting synthetic datasets. • Empirically demonstrates that training on structurally complex sentences yields comparable or superior MT performance (especially in low-resource contexts) while reducing training dataset requirements. • Presents a straightforward yet effective method (the LALITA score) for assessing structural complexity. • In situations where the expense of developing and training MT data is high, LALITA provides an effective approach