Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases

Rena Gao ♡ , Xuetong Wu ♡ * , Tatsuki Kuribayashi ♢ , Mingrui Ye ♣ , Siya Qi ♣ Carsten Roever ♡ , Yuanxing Liu ♠ , Zheng Yuan ♣ , Jey Han Lau ♡ ♡ The University of Melbourne ♣ King’s College London ♢ MBZUAI ♠ Harbin Institute of Technology {rena.gao,carsten}@unimelb.edu.au, {wfyitf,jeyhan.lau}@gmail · 2025

Read Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases by Rena Gao ♡ , Xuetong Wu ♡ * , Tatsuki Kuribayashi ♢ , Mingrui Ye ♣ , Siya Qi ♣ Carsten Roever ♡ , Yuanxing Liu ♠ , Zheng Yuan ♣ , Jey Han Lau ♡ ♡ The University of Melbourne ♣ King’s College London ♢ MBZUAI ♠ Harbin Institute of Technology {rena.gao,carsten}@unimelb.edu.au, {wfyitf,jeyhan.lau}@gmail (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14507.txt.

Introduction

People learning English carry habits from their first language into their second, and teachers know the patterns well. Rena Gao and co-authors ask whether large language models can imitate those patterns on request. In interview-style dialogues, they prompt models to play learners from seven first languages, then compare the output with real learner data using information measures and distribution checks.

The modern models they tested, including Qwen2.5, LLaMA 3.3, DeepSeek-V3, and GPT-4o, did reproduce many first-language habits. Japanese, Korean, and Mandarin backgrounds bent tense agreement in the simulated speech, while an Urdu background shaped noun and verb pairings. The authors read this as evidence that models can stand in for learners when building practice dialogues or grading tools. They frame the work as a step toward language teaching software that reflects how real learners actually speak.

Catalog record

CREATORRena Gao ♡ , Xuetong Wu ♡ * , Tatsuki Kuribayashi ♢ , Mingrui Ye ♣ , Siya Qi ♣ Carsten Roever ♡ , Yuanxing Liu ♠ , Zheng Yuan ♣ , Jey Han Lau ♡ ♡ The University of Melbourne ♣ King’s College London ♢ MBZUAI ♠ Harbin Institute of Technology {rena.gao,carsten}@unimelb.edu.au, {wfyitf,jeyhan.lau}@gmail
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14507

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Simulate L2-English Dialogue? An Information-Theoretic Analysis of L1-Dependent Biases Rena Gao ♡ , Xuetong Wu ♡ * , Tatsuki Kuribayashi ♢ , Mingrui Ye ♣ , Siya Qi ♣ Carsten Roever ♡ , Yuanxing Liu ♠ , Zheng Yuan ♣ , Jey Han Lau ♡ ♡ The University of Melbourne ♣ King’s College London ♢ MBZUAI ♠ Harbin Institute of Technology {rena.gao,carsten}@unimelb.edu.au, {wfyitf,jeyhan.lau}@gmail.com {tatsuki.kuribayashi}@mbzuai.ac.ae , {yxliu}@ir.hit.edu.cn {mingrui.ye, siya.qi, zheng.yuan}@kcl.ac.uk Abstract This study evaluates Large Language Models’ (LLMs) ability to simulate non-native-like English use observed in human second language (L2) learners interfered with by their native first language (L1). In dialogue-based interviews, we prompt LLMs to mimic L2 English learners with specific L1s (e.g., Japanese, Thai, Urdu) across seven languages, comparing their outputs to real L2 learner data. Our analysis examines L1-driven linguistic biases, such as reference word usage and avoidance behaviors, using information-theoretic and distributional density measures. Results show that modern LLMs (e.g., Qwen2.5, LLAMA3, DeepseekV3, GPT 4o) replicate L1-dependent patterns observed in human L2 data, with distinct influences from various languages (e.g., Japanese, Korean, and Mandarin significantly affect tense agreement, and Urdu influences noun-verb collocations). Our results reveal LLMs’ potential for L2 dialogue generation and evaluation for future educational applications. 1 Introduction The widespread use of Large Language Models (LLMs) in language communication and education has opened opportunities to study their ability to simulate human-like language, particularly in second language (L2) communication Liang et al. (2024) ; Cherednichenko et al. (2024) , as illustrated by Figure 1 . Such an L2-speaker simulation will be helpful for, e.g., predicting L2 speakers’ biases in a pedagogical situation Settles et al. (2018) , developing an L2-speaking agent Timpe-Laughlin et al. (2022) , virtual language-learning applications Bibauw et al. (2022) , emulating diverse agents to simulate the diversity of L2 speakers in real world Ge et al. (2024) , and potentially assessing LLMs’ cognitive plausibility from a cross-lingual perspective Aoyama and Schneider (2024a) . However, the ability of LLMs to accurately replicate linguistic patterns of non-native speakers and the systematic influence of L1 knowledge on L2 generation remain underexplored Chen et al. (2024) , especially in the dialogue domain Veivo and Mutta (2025) and in non-native contexts Fincham and Alvarez (2024) . This leads us to ask: Can LLMs effectively mimic human-like dialogue performance in L2 contexts? Figure 1: Examples of L2 English dialogue from human speakers, which can generally be biased by their native L1 knowledge, e.g., with particular errors . To address this question, it is crucial to understand the role of native linguistic knowledge in areas such as language education and cross-lingual communication Levenston (1971) ; Schachter (1974) ; Kleinmann (1977) ; Brooke and Hirst (2012) . L2 speakers’ use of English is often influenced by their L1 traits Takahashi (2024) , especially for Asian native speakers whose first language (L1) is diverse from English Pan (2024) , resulting in distinct L1-L2 linguistic patterns Bailey et al. (2021) , including grammatical constructions and lexical choices in spoken dialogues Aoyama and Schneider (2024b) ; Downey et al. (2023) . To investigate whether LLMs simulate similar patterns, we propose an information-theoretic evaluation framework grounded in multiple linguistic perspectives: key features from grammatical/semantic accuracy, fluency, discourse-level cohesion, and pragmatics that shape the communicative outcome Schwandt (2001) ; Sun et al. (2021) ; Gao and Wang () ; Santiago-Garabieta et al. (2023) . By analyzing these aspects, we explore how accurately various LLMs simulate L2-like dialogues that align with human linguistic behaviors across different L1 backgrounds. 1 1 1 All codes and dataset can be found https://github.com/RenaGao/LLMPirorknowledge For benchmark data, we utilize the ICNALE dataset Ishikawa (2023) , which includes recordings from 435 human L2 speakers with 18 L1s and manual transcripts comprising approximately 1.6M tokens. An information-theoretic analysis is applied to evaluate LLMs’ L1-dependent biases by comparing LLM-generated dialogues (with a prompt to simulate L2 English with a specific L1) with human counterparts. To address challenges in reflecting L1 background in LLMs’ L2 generation, as an initial foray, we employ native knowledge injection prompting (e.g., Simulate L2 English dialogue spoken by Japanese based on provided Japanese native linguistic knowledge ) Dong et al. (2022) ; Santiago-Garabieta et al. (2023) ; Bibauw et al. (2022) . Through our exploration, we demonstrate that simple L1 prompting has a significant impact on LLM-generated L2 dialogues. For example, Japanese, Korean, and Mandarin L1 influence tense agreement, Thai and Malay L1 affect speech acts, while Urdu L1 impacts noun-verb collocations. Our information-theoretic evaluation quantifies their human-like output, which is further supported by qualitative analysis. Ultimately, our study paves the way for using LLMs to simulate human L2 dialogues. Summarizing our contributions: • We propose a new evaluation framework with eight linguistic features, covering grammatical/semantic accuracy, fluency, cohesion, and pragmatics perspectives, designed to evaluate the impact of L1 information on LLM-generated dialogues. This framework enables systematic analysis of how native language traits (in humans/LLMs) shape linguistic features in cross-lingual dialogue generation. • We further propose an information-theoretic metric to quantify L1 influence on LLM dialogue generation, revealing L1-dependent differences such as reference word , modifiers and numerals usages. • We show that, through prompting, LLMs can generate dialogues with varying degrees of non-native-like linguistic features influenced by different L1s, paving a new way for LLMs to simulate L2 communications. 2 Related Work 2.1 Bilingual Knowledge for LLMs L1 interference in humans and LMs Native language profoundly influences L2 language use in humans Levenston (1971) ; Schachter (1974) ; Kleinmann (1977) ; Brooke and Hirst (2013) . This language interference effect biases, for example, the syntactic constructions Felker et al. (2021) and discourse flows Bailey et al. (2021) in L2, and the dialogue patterns are not an exception Veivo and Mutta (2025) . When it comes to neural LMs, the cross-lingual transferability of LMs and their human-likeness has also gained attention, but prior studies have exclusively focused on sentence-level evaluations Oba et al. (2023) ; Yadavalli et al. (2023) ; Elshin et al. (2024) . Such perspectives can easily be extended to the dialogue level, involving discourse-level cohesion/coherence and L1-dependent, nuanced differences in dialogue strategy Abe and Roever (2019) ; Gao et al. (2024) . Moreover, LLMs are now deployed to generate dialogue (e.g., chat interactions); evaluating their ability in a dialogue scenario generally aligns with their practical usage Jin et al. (2024) ; Veivo and Mutta (2025) . That said, our scope is limited to just simulating L2-like language use in a behavioral sense; LM’s cognitive plausibility as an L2 learner, while interesting and related, is beyond of the scope of this paper. Bilingual Knowledge in LLMs Bilingual knowledge typically impacts LLM in cross-lingual and multilingual tasks Miah et al. (2024) . For example, leveraging shared grammatical features, bilingual LLM excels with typologically similar language pairs like English-Spanish, improving coherence and fluency through transfer learning Jeon and Van Roy (2022) . On the other hand, handling distant cross-lingual pairs, such as English-Chinese, poses challenges (i.e., negative language transfer) due to differences in their grammatical features such as word order Ranaldi and Pucci (2023) , requiring targeted training and alignment of grammatical constructs Přibáň et al. (2024) . In the context of dialogue tasks, limited L2 dialogue data

Read the complete text