Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties

Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, Edward Choi · 2025

Read Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties by Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, Edward Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20875.txt.

Introduction

Jiyoung Lee and six co-authors build a test rig for how language models handle the many Englishes of the world in a 2025 paper. Models are graded almost wholly on standard American English. Users who write in other varieties may get a worse tool without anyone noticing.

Their rig rewrites standard test sets into 38 varieties of English, guided by expert rules drawn from language studies plus model help. Six test sets were recast and seven top models graded on each. Scores fell hard on some varieties, by up to about 46 percent. The authors call for wider grading as a fairness must.

Catalog record

CREATORJiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, Edward Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20875

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties Jiyoung Lee, Seungho Kim, Jieun Han, Jun-Min Lee, Kitaek Kim, Alice Oh, Edward Choi Abstract Large Language Models (LLMs) are predominantly evaluated on Standard American English (SAE), often overlooking the diversity of global English varieties. This narrow focus may raise fairness concerns as degraded performance on non-standard varieties can lead to unequal benefits for users worldwide. Therefore, it is critical to extensively evaluate the linguistic robustness of LLMs on multiple non-standard English varieties. We introduce Trans-EnV, a framework that automatically transforms SAE datasets into multiple English varieties to evaluate the linguistic robustness. Our framework combines (1) linguistics expert knowledge to curate variety-specific features and transformation guidelines from linguistic literature and corpora, and (2) LLM-based transformations to ensure both linguistic validity and scalability. Using Trans-EnV, we transform six benchmark datasets into 38 English varieties and evaluate seven state-of-the-art LLMs. Our results reveal significant performance disparities, with accuracy decreasing by up to 46.3% on non-standard varieties. These findings highlight the importance of comprehensive linguistic robustness evaluation across diverse English varieties. Each construction of Trans-EnV was validated through rigorous statistical testing and consultation with a researcher in the field of second language acquisition, ensuring its linguistic validity. Our code and datasets are publicly available. 1 1 1 Code: https://github.com/jiyounglee-0523/TransEnV Dataset: https://huggingface.co/collections/jiyounglee0523/transenv-681eadb3c0c8cf363b363fb1 1 Introduction Large Language Models (LLMs) [ 1 , 60 , 24 ] have shown impressive performance, even surpassing humans on several tasks [ 41 , 31 , 21 ] . However, most evaluation benchmarks are written in Standard American English (SAE), overlooking the rich diversity of English varieties. English is spoken in a wide range of varieties , including regional dialects and forms used by non-native speakers [ 8 ] . This narrow linguistic focus may raise fairness concerns, as LLMs tend to underperform on non-standard varieties [ 71 , 6 ] , potentially leading to unequal benefits for global users. Therefore, assessing LLM performance in multiple English varieties is essential to ensure its fairness. While several datasets have been introduced to evaluate the robustness of LLMs to varieties [ 7 , 14 , 54 , 33 ] , they remain limited in size, variety coverage, and task diversity, making them inadequate for comprehensive evaluation. To rigorously assess the linguistic robustness of LLMs, it is necessary to evaluate on existing benchmark datasets across a diverse range of English varieties. This, in turn, requires a framework capable of automatically converting SAE benchmarks into any desired target variety. Although prior studies have proposed such transformations [ 38 , 53 , 25 , 72 , 73 ] , these approaches often suffer from scalability challenges, lack of expert knowledge, or failure to adequately capture linguistic diversity. To this end, we introduce Trans-EnV , a framework that automatically transforms SAE datasets into a desired target variety. We focus on two widely studied types of variety: regional dialect and English as a Second Language (ESL) English [ 30 , 5 , 56 ] . Regional dialects, henceforth called as dialects , refer to geographically localized varieties of English, such as Scottish or Irish English. ESL English refers to language produced by speakers whose first language is not English. We begin by collecting linguistic features from expert-curated resources and large-scale ESL corpora to ensure rigorous and accurate information. Then, for each feature, we create transformation guidelines that specify the steps to apply the feature to an SAE sentence. Then we use an LLM to transform SAE sentences by following the guidelines. An overview of Trans-EnV is provided in Figure 1 . LLMs are used both for the guideline generation and the sentence transformation, making Trans-EnV both labor-efficient and scalable across datasets. To ensure linguistic validity, we consulted a researcher in the field of second language acquisition throughout the entire development, aligning our methodology with established linguistic theory and practice. We translate six widely used benchmark datasets into 38 varieties consisting of 18 dialects and 20 ESL English varieties. We evaluate seven state-of-the-art LLMs on the transformed benchmarks and observe that model performance generally degrades across most varieties, with particularly pronounced drops in ESL English (12.5% and 46.3% performance drop at maximum for each dialect and ESL English). In addition, we find that linguistic robustness is notably weaker on tasks that require reasoning. Models specialized in reasoning tend to be more robust than others, suggesting that strong reasoning capabilities may contribute to improved robustness. Our contributions are summarized as follows: • We introduce Trans-EnV, a framework that automatically transforms SAE-written datasets into specified target English varieties. By leveraging an LLM for transformation, our framework is both labor-efficient and scalable across multiple datasets. • Trans-EnV is grounded in expert-curated linguistic resources, and its construction was validated through rigorous statistical testing and consultation with a researcher in the field of second language acquisition, ensuring its linguistic validity. • We conduct extensive experiments by transforming six widely used benchmark datasets into 38 varieties and evaluating seven LLMs. The results show that LLMs exhibit notable weaknesses in handling non-standard English varieties, particularly in ESL English. 2 Related Work English Variety Disparity in LLMs. English variety refers to various forms of English used across different regions, communities, or learner groups, including both Standard American English and non-standard forms. The non-standard forms include regional dialects ( e . g ., Scottish English) and ESL learner usages ( e . g ., Arabic English) [ 42 , 26 ] . Despite such diversity in English, LLMs do not perform well particularly in non-standard English in the form of dialects [ 32 , 58 , 11 , 66 , 36 ] and ESL learners’ usages [ 43 , 37 ] . For instance, in non-SAE settings, LLMs underperform in tasks such as language generation and understanding [ 11 , 25 , 58 , 43 ] , reasoning capability [ 38 , 70 ] and instruction following [ 16 ] . In addition, LLMs respond in a more stereotyping, demeaning, unnatural, and condescending manner to under-represented varieties [ 16 , 28 , 37 ] . Zhou 2020 showed that LLMs have shown persistent biases against non-SAE particularly in tasks involving toxicity. English Variety Dataset. Previous work on transforming datasets for varieties typically uses one of three methods: (i) manual curation [ 38 , 53 ] , (ii) LLM generation [ 25 ] , and (iii) rule-based transformation [ 72 , 73 ] . The first approach involves humans manually creating the entire dataset. While this method can produce high-quality results, recruiting qualified human annotators is resource-intensive, and scaling to multiple English varieties and datasets is challenging. The second approach relies entirely on LLMs to generate varieties. While this method is scalable and convenient, several studies have highlighted the limitations of LLMs in accurately reproducing under-represented varieties of English [ 43 , 64 , 4 , 15 ] , underscoring the risk of relying solely on LLMs. The third approach employs deterministic, rule-based transformations to transform SAE sentences into targeted varieties. However, this method demands substantial human effort to manually craft transformation rules for each linguistic feature. Moreover, it falls short in capturing the full spectrum of linguistic variation, including lexical choices and pragmatic nuances.

Read the complete text