DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models
Jinxiang Xie, Yilin Li, Xunjian Yin, Xiaojun Wan · 2024
Read DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models by Jinxiang Xie, Yilin Li, Xunjian Yin, Xiaojun Wan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12832.txt.
Introduction
Jinxiang Xie and co-authors propose in 2024 DSGram, a new way to grade grammar correction systems in the age of large models. Model fixes often differ from the one gold answer, so old match-based scores can mislead. A fair grade needs more than overlap with a single key.
DSGram scores meaning flow, edit size, and fluency, with weights that shift by case. A ranking method joined with large models sets how much each part counts. The team built a data set with human grades and model-made sentences to test the frame. Results show sharper and more fair grades than past scores give.
Catalog record
| CREATOR | Jinxiang Xie, Yilin Li, Xunjian Yin, Xiaojun Wan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models Jinxiang Xie, Yilin Li, Xunjian Yin, Xiaojun Wan Abstract Evaluating the performance of Grammatical Error Correction (GEC) models has become increasingly challenging, as large language model (LLM)-based GEC systems often produce corrections that diverge from provided gold references. This discrepancy undermines the reliability of traditional reference-based evaluation metrics. In this study, we propose a novel evaluation framework for GEC models, DSGram, integrating Semantic Coherence, Edit Level, and Fluency, and utilizing a dynamic weighting mechanism. Our framework employs the Analytic Hierarchy Process (AHP) in conjunction with large language models to ascertain the relative importance of various evaluation criteria. Additionally, we develop a dataset incorporating human annotations and LLM-simulated sentences to validate our algorithms and fine-tune more cost-effective models. Experimental results indicate that our proposed approach enhances the effectiveness of GEC model evaluations. Code — https://github.com/jxtse/GEC-Metrics-DSGram Datasets — https://huggingface.co/datasets/jxtse/DSGram 1 Introduction Grammatical Error Correction (GEC) models aims to automatically correct grammatical errors in natural language texts, enhancing the quality and accuracy of written content. Traditionally, the evaluation of GEC models has employed a variety of metrics, categorized into those requiring a reference (reference-based evaluation) and those that do not (reference-free evaluation). Reference-based metrics such as BLEU ( Papineni et al. 2002 ) , ERRANT ( Bryant et al. 2017 ) , and M² ( Dahlmeier et al. 2012 ) compare the model-generated text with a correct reference text to evaluate the accuracy of grammatical corrections, and they are widely used in this area. Despite the usefulness of these metrics, they possess inherent limitations. For example, the golden reference may not encompass all potential corrections ( Choshen et al. 2018 ) , and the alignment of existing automatic evaluation metrics with human judgment is often weak ( Coyne et al. 2023 ) . Additionally, LLMs-based GEC models may excessively correct sentences, resulting in unnecessary editing not captured by traditional metrics ( Fang et al. 2023b ) . Figure 1: Running examples and evaluation results of several existing metrics. The sentence is from Fang et al. 2023b . This figure illustration presents the outcomes of an input sentence processed through two representative GEC models and also the corresponding reference. The metrics’ scores are placed under the output. The highlighted in blue represents the over-correction. The highlighted in red indicates poor fluency. Notably, BLEU fails to differentiate between over- and under-correction, whereas SOME cannot capture over-correction. Figure 2: Architecture of the DSGram method. It begins with the input of sentence pairs (original and corrected). Large language models (such as GPT-4 and LLaMA3) are employed to generate dynamic weights, which are further refined through a judgment matrix and a consistency check. These dynamic weights are normalized and then used to score sub-metrics. The final output consists of both the calculated weights and the overall score for the evaluated correction. Conversely, reference-free metrics like Perplexity, GLEU ( Napoles et al. 2015 ) , and SOME ( Yoshimura et al. 2020 ) assess the quality of generated text directly, proving beneficial when a correct reference text is unavailable, or when a single (or several) reference is insufficient for GEC evaluation in the era of LLMs. While existing reference-free evaluation metrics like SOME offer an analytical foundation, they no longer fully encompass the scope of GEC evaluation. It is necessary to design new sub-metrics based on the original sub-metrics and emerging requirements. Figure 1 shows running examples of representative GEC systems and the limitations of current evaluation metrics. This study seeks to overcome these limitations by introducing a novel evaluation framework, DSGram, for GEC models. DSGram integrates Semantic Coherence, Edit Level, and Fluency, and determines the appropriate weights for these criteria. By adjusting these weights based on the evaluation scenario’s context, a more nuanced and context-sensitive evaluation approach can be developed. Our contributions are summarized as follows: • We introduce new sub-metrics for GEC evaluation, diverging from previous categorical approaches. Our metrics optimize past sub-metrics by adding an evaluation of over-editing. • We propose a novel dynamic weighting-based GEC evaluation method, DSGram, which integrates the Analytic Hierarchy Process ( Ana 1987 ) with large language models to ascertain the relative importance of different evaluation criteria. • We present two datasets: DSGram-Eval, created through human scoring, and DSGram-LLMs, a larger dataset designed to simulate human scoring for fine-tuning. Both datasets utilize sentences from the CoNLL-2014 and BEA-2019 test sets to facilitate rigorous evaluation. 2 Related Work Numerous studies have focused on evaluating GEC models. This section provides an overview of key research utilizing LLMs for GEC assessment. 2.1 Model-Based Evaluation Metrics Model-based evaluation metrics have garnered significant attention, especially in the domain of GEC. BLEURT ( Sellam et al. 2020 ) is a versatile metric that evaluates based on (prediction, reference) pairs. It employs a scalable pretraining phase where it learns to predict automatically generated signals of supervision from semantically comparable synthetic pairs. A burgeoning trend in automatic evaluation involves the direct application of LLMs for assessment purposes. Liu et al. 2023 have LLMs generate numerical ratings by interpreting descriptions of evaluation criteria through a Chain-of-Thought method ( Wei et al. 2023 ) . Sottana et al. 2023 demonstrate the viability of GPT-4 in GEC assessment, highlighting an approach that uses natural language instructions to define evaluation criteria. 2.2 Reference-Free Evaluation Reference-free evaluation is a method of assessing the performance of models without relying on reference. Asano et al. 2017 integrate three sub-metrics, which are Grammaticality, Fluency, and Meaning Preservation, to surpass reference-based metrics. They employ a language model and edit-level metrics as Fluency and Meaning Preservation sub-metrics, respectively, although these sub-metrics are not tailored for manual evaluation. The final score is determined through a weighted linear summation of each individual evaluation score. SOME is a reference-free GEC metric that follows the approach of Asano et al. 2017 . It utilizes three distinct BERT models, each dedicates to one scoring aspect. The researchers construct a novel dataset for training these BERT models by annotating the outputs from various GEC systems on the CoNLL-2013 test set across the three scoring dimensions. However, in real-world applications, the use of just three evaluation metrics presents a limitation, as it hinders the generation of a holistic score in an inherently intuitive way. Consequently, this constraint diminishes its effectiveness in directing the training of GEC models. In the sphere of GEC evaluation, Wu et al. 2023 assessed ChatGPT and observed a tendency towards over-correction, which might be attributed to its extensive generative capacities as a large language model. Similarly, Fang et al. 2023b noted that ChatGPT makes over-corrections, leading to revised sentences with high fluency. These findings align with the outcomes of our own experiments, which indicate that existing reference-free metrics do not adequately reflect these issues. 2.3 GEC Dataset with Human Scoring There are very few GEC datasets with human evaluation scores. The dataset annotated by Yoshimura et al. 2020 includes individual scores for Grammaticality, Fluency, and Meaning Preservation, but lacks an overall score. GJG15 ( Grundkiewicz et al. 2015 ) and SEEDA ( Kobayashi et al. 2024 ) are manually annotated but provide ranking information rather than scores. Sottana et al. 2023 conducted ratings based on the gold