Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation

Peerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, Potsawee Manakul · 2025

Read Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation by Peerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, Potsawee Manakul (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05898.txt.

Introduction

Peerat Limkonchotiwat and co-authors test large language models on Thai dialects in a 2025 paper. Models do well on many language tasks, yet their skill in less common languages is less clear. Most benchmarks cover only the main form of a language. Local dialects are often left out.

The team builds a benchmark for Northern, Northeastern, and Southern Thai. It covers summary writing, questions, translation, chat, and food tasks. They also set rules for human grading of fluency and dialect accuracy. Model scores fell sharply on local dialects versus standard Thai, they report. Only some closed commercial models kept some fluency.

Catalog record

CREATORPeerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, Potsawee Manakul
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05898

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation Peerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, Potsawee Manakul Abstract Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs’ ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency 1 1 1 Data and Evaluation Tool: https://github.com/mrpeerat/Thai_local_benchmark . † † * Equal contributions 1 Introduction Large language models (LLMs) play a crucial role in natural language processing (NLP) by significantly enhancing downstream performance across diverse tasks and languages. Studies have shown that LLMs surpass traditional multilingual models Devlin et al. (2019) ; Conneau et al. (2020) in tasks such as question answering and summarization Touvron et al. (2023) ; Pipatanakul et al. (2023) . One key reason is that LLMs have significantly larger parameter counts and training datasets, often exceeding traditional models by a factor of 26, from 0.3 billion to 8 billion parameters. As a result, LLMs demonstrate greater robustness than traditional models across most scenarios, particularly in low-resource languages such as Thai Pengpun et al. (2024) ; Phatthiyaphaibun et al. (2024) , Indonesian Cahyawijaya et al. (2024) ; Cahyawijaya et al. (2023) , and Tagalog Montalan et al. (2024) ; Gamboa and Lee (2025) . Figure 1: (a) We showed some unique characteristics of Thai local dialects compared to Central Thai: 1. shared tokens in local dialects; 2. unique words for each local dialect; and 3. WH- question tokens that are different in each local dialect. (b) Illustration of our local dialect evaluation metric. The example demonstrates the assessment of generation and fluency, highlighting variations in spelling and pronunciation among different Thai dialects. Researchers are increasingly evaluating LLMs in underrepresented languages, particularly Thai, to assess their robustness and consistency in low-resource languages. Thai-H6 Kim et al. (2025) introduces an evaluation benchmark for assessing LLMs’ understanding of Thai cultural knowledge. SEACrowd Lovenia et al. (2024) introduces a Southeast Asian benchmark for evaluating LLMs on local and cross-lingual texts, audios, and images, focusing on natural language generation and understanding. The Thai LLM benchmark 2 2 2 https://huggingface.co/spaces/ThaiLLM-Leaderboard/leaderboard extends SEACrowd by incorporating LLM-as-a-Judge and Thai Exam datasets for a Thai language evaluation. However, these benchmarks primarily focus on the Central Thai dialect. Therefore, challenges observed in the Central dialect may also reflect underlying issues in local dialects. Although there are 69 million Thai speakers, only about 20 million have Central Thai as their first language. The rest primarily grow up speaking other dialects, including approximately 15.2 million who speak Northeastern Thai (Isan), 6 million who speak Northern Thai (Lanna), and 4.5 million who speak Southern Thai (Dambro) or other local varieties. As shown in Figure 1 a, a distinct feature of Thai local dialects is the blending of words between Central Thai and local dialects, as well as among different local dialects. Moreover, each local dialect has distinct words, some unique to that dialect and others shared with other dialects but not found in others. Recent studies examine the evaluation of fluency and understanding in local dialects of Indonesian Syarfina et al. (2022) ; Cahyawijaya et al. (2023) ; Veria Septianingtias and Ariyani (2024) and Filipino Tabajunda (2018) ; Guevara et al. (2024) . Evaluating the fluency and understanding of LLMs in Thai local dialects remains underexplored. In this paper, we introduce a benchmark for evaluating local Thai dialects–Isan, Lanna, and Dambro–to assess LLMs’ understanding of these dialects. Specifically, we evaluate LLMs on five tasks – summarization, question answering, translation, conversation, and food – using instructions, inputs, and labels written in local dialects. Additionally, we introduce human evaluation alongside a new Thai local dialect guideline and metric to assess LLMs’ capabilities in understanding and generating local dialects. The new metric evaluates the generation and fluency of local dialects, rewarding or penalizing the model for fluency in a manner consistent with human speakers. Our experimental results show that LLM performance significantly declines on summarization and translation tasks when inputs are written in local dialects, compared to inputs in Central Thai. Furthermore, only proprietary models–GPT and Gemini–are capable of speaking local Thai dialects, as shown by both human evaluation and our new metric. Our contributions are as follows: We introduce a benchmark for evaluating three Thai local dialects. We also propose a guideline and metric to assess the generation and fluency of LLMs in Thai local dialects. Lastly, we are the first to investigate Thai local dialects in the context of LLMs, presenting comprehensive evaluations using both traditional evaluation metrics and human assessment. 2 Benchmark and Metric 2.1 Benchmark Formulation Data Selection. We create a new benchmark for evaluating three Thai local dialects–Isan, Lanna, and Dambro–drawing on examples from the Thai LLM leaderboard to assess model performance. We select generation tasks—Question Answering (QA), summarization, and translation—because they are more complex and better suited for assessing model knowledge than more straightforward multiple-choice QA tasks. We randomly selected 20 samples for each task, resulting in 80 samples per dialect, where the translation task was both English to local dialects and vice versa. In addition, we include 10 topics on food (i.e., asking for a food recipe of [FOOD_NAME] ) and conversations between two individuals discussing Thai traditional and cultural topics (i.e., talking bout [TOPIC_NAME] ). The full topics can be found in Appendix A.7 . Gold Standard. Our dataset uses the gold standard provided by the original Central Thai examples. Note that we do not create labels for the food and conversation topics, as these tasks are open-ended and lack a definitive correct answer. Instead, we rely on human evaluation to assess these tasks. The number of samples is 100, all written in Central Thai. Translation. We enlisted native speakers of each local dialect to translate the 100 samples from Central Thai into their respective dialects. This includes translating the input, context (for QA and summarization tasks), prompts, and labels, except for those related to food and conversation samples. Furthermore, all annotators are asked to review translations completed by their peers within the same region to ensure quality and consistency. Therefore, the total number of samples is 400, comparable to previous LLM-as-a-Judge and Human evaluation benchmarks Zheng et al. (2023) ; Qin et al. (2024) . For the full information of data statistics, prompts, topics, annotator demographics, and guidelines, see Appendix A . Model Dialect QA \Delta Diff Summarization \Delta Diff Translation (E-L) \Delta Diff Translation (L-E) \Delta Diff Llama-3.1-8b Central 60.74 - 20.07 - 11.41 - 44.61 - Typhoon1.5-8b Central 33.36 - 11.52 - 5.14 - 47.35 - Llama-3.1-70b Central 56.44 - 23.03 - 18.11 - 51.07 - Typhoon1.5-70b Central 46.99 - 10.83 -

Read the complete text