Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation

2025

Read Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06680.txt.

Introduction

Keunhyeung Park, Seunguk Yu, and Youngbin Kim steer language models toward genuine Korean local dialect instead of lightly edited standard speech. Translating into dialect is hard because models drift back to the standard form, and word-overlap metrics reward that copying while punishing real dialect. Evaluation itself was part of the problem.

Their DIA-REFINE framework loops translation, verification by dialect classifiers, and feedback until the output sounds faithful. Two new scores measure the linguistic shift achieved and the share of output truly in dialect, exposing fake successes where overlap looks high but dialect is absent. Across Korean dialects the loop improved fidelity over zero-shot and example-guided baselines, with responsiveness varying by model.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06680

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation Abstract Standard-to-dialect machine translation remains challenging due to a persistent dialect gap in large language models and evaluation distortions inherent in n-gram metrics, which favor source copying over authentic dialect translation. In this paper, we propose the dialect refinement ( DIA-REFINE ) framework, which guides LLMs toward faithful target dialect outputs through an iterative loop of translation, verification, and feedback using external dialect classifiers. To address the limitations of n-gram-based metrics, we introduce the dialect fidelity score ( DFS ) to quantify linguistic shift and the target dialect ratio ( TDR ) to measure the success of dialect translation. Experiments on Korean dialects across zero-shot and in-context learning baselines demonstrate that DIA-REFINE consistently enhances dialect fidelity. The proposed metrics distinguish between False Success cases, where high n-gram scores obscure failures in dialectal translation, and True Attempt cases, where genuine attempts at dialectal translation yield low n-gram scores. We also observed that models exhibit varying degrees of responsiveness to the framework, and that integrating in-context examples further improves the translation of dialectal expressions. Our work establishes a robust framework for goal-directed, inclusive dialect translation, providing both rigorous evaluation and critical insights into model performance. Keywords: Korean Dialect Translation, Iterative Refinement, Dialectal Evaluation Bias Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation Keunhyeung Park, Seunguk Yu, Youngbin Kim Chung-Ang University [email protected], [email protected], [email protected] Abstract content 1. Introduction Recent advancements in large language models (LLMs) have revolutionized the field of machine translation ( 17 ) , highlighting the importance of inclusive approaches for diverse low-resource languages ( 7 ) . In this context, dialect research holds significant academic and social value, as it contributes to preserving the linguistic heritage of potentially marginalized regional languages and enhancing technological inclusivity ( 31 ) . Despite the progress of LLMs, a significant performance disparity persists between standard and non-standard dialects ( 10 ; 11 ; 13 ) . This behavior exhibits an asymmetry depending on the translation direction, with translating from a standard language to a dialect being more challenging than the reverse ( 30 ; 24 ) . The difficulty of LLMs in translating distinctive dialectal features is evidenced in our experimental results, where some models achieved a zero-shot success rate of only 2%, with most outputs closely resembling the standard language. While dialect machine translation exhibits these limitations, research on controlling and evaluating the dialectal fidelity of translation outputs remains scarce. To address this gap, we propose a framework that utilizes external feedback from dialect classifiers to guide LLMs toward translating target dialect outputs, as illustrated in Figure 1 . Our core methodology, dialect refinement ( DIA-REFINE ), verifies whether the output possesses the linguistic features of the target dialect based on the dialect classifier’s prediction. If the output fails to satisfy the target dialect condition, explicit feedback derived from the classification result is provided to the model for re-translation. Through this iterative process, the LLM can be effectively controlled to perform goal-oriented dialect translation. Figure 1: An overview of our dialect refinement ( DIA-REFINE ) framework. The LLM’s output is verified by an external ensemble of dialect classifiers, which provides explicit feedback to guide the model toward translating the target dialect. Variants Forms Analysis / Key Features BLEU chrF++ DFS Source (Standard) 제주도는 특별한 약초들이 많이 없으니까 그냥 그냥 넘긴 거 같아 Standard grammar and vocabulary. – – – Reference ( Jeju Dialect) 제주도는 특별한 약초들이 많이 엇이난 그냥 그냥 넘긴 거 닮아 Contains target dialectal features. – – – Hypothesis 1 제주도 엔 특별한 약초 덜 이 많이 읎응께 그냥 그냥 넘긴 거 같아 True Attempt : Generates multiple dialectal features ( -엔 (-en), -덜 (-deol), -읎응께 (-eup-eung-kke)) 27.78 36.80 0.672 Hypothesis 2 제주도는 특별한 약초들이 많이 없으니깐 그냥 그냥 넘긴 거 같아 False Success : Output mirrors the standard source rather than the intended dialect. 52.54 67.82 -0.672 Table 1: Evaluation distortions in Korean dialect translation: while n-gram metrics favor surface overlap ( False Success ), the proposed DFS metric accurately captures the shift toward the dialectal output ( True Attempt ). Underlined morphemes denote dialectal expressions in the hypothesis. The shared meaning is ‘ I think they just passed on Jeju Island since it doesn’t have a lot of special medicinal herbs there. ’ However, traditional n-gram-based metrics such as BLEU ( 23 ) and chrF++ ( 26 ) have been repeatedly reported to have a structural blind spot ( 1 ; 6 ) . This vulnerability can lead to higher scores for False Success , which involves merely copying the source, than for a True Attempt to generate dialectal features 1 1 1 In this paper, we define False Success as cases exhibiting relatively high n-gram but low DFS/TDR scores, and True Attempt as cases showing the opposite pattern, with low n-gram and high DFS/TDR scores, within our comparative analysis across models. . As shown in Table 1 , this paradox causes outputs that merely copy the standard source being evaluated more favorably than genuine attempts to generate dialectal features. This biased evaluation system hinders the accurate assessment of practical gains of the dialectal translation outputs. To address the limitations of existing metrics and rigorously validate our framework, we introduce a new multifaceted evaluation system. It comprises the dialect fidelity score ( DFS ), which quantifies the linguistic shift of translated outputs toward the dialectal reference, and the target dialect ratio ( TDR ), which measures the proportion of outputs correctly classified as the target dialect. These metrics are designed to reliably assess the effectiveness of our framework in steering translation toward the target dialect. To validate the proposed framework, we conducted experiments translating standard Korean into three major dialects with distinct segmental features, Jeolla , Gyeongsang , and Jeju , two of which have been largely unaddressed in previous research. Using several state-of-the-art LLMs, we performed a comparative analysis of existing approaches, including zero-shot and in-context learning, against our proposed DIA-REFINE framework to evaluate model performance under various conditions. The main contributions of this study are as follows: (i) We propose the dialect refinement ( DIA-REFINE ) framework, an effective mechanism for controlling dialect translation. (ii) We introduce a robust evaluation system with DFS and TDR metrics, addressing the limitations of n-gram-based metrics and enabling an accurate assessment of dialectal fidelity. (iii) Our analysis demonstrates that the proposed metrics effectively distinguish between cases of False Success and True Attempt across models. Moreover, we establish that combining our framework with in-context examples yields the most effective performance in dialect translation. This paper is organized as follows: Section 2 reviews related work on recent trends in dialect research, steering generation and challenges in dialect evaluation. Section 3 and 4 present the DIA-REFINE framework and experimental setup, respectively. Section 5 details our evaluation metrics. Section 6 reports the main results and our detailed analysis, and Section 7 concludes. We plan to release the code to facilitate the proposed framework and metrics, thereby supporting further research in this area 2 2 2 Due to the source dataset’s terms of use, the processed dataset cannot be released; however, we made our code publicly available to

Read the complete text