Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction
Takumi Goto, Justin Vasselli, Taro Watanabe · 2024
Read Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction by Takumi Goto, Justin Vasselli, Taro Watanabe (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13110.txt.
Introduction
Takumi Goto, Justin Vasselli, and Taro Watanabe made grammar correction scores explainable in twenty twenty-four by crediting individual edits. Sentence-level metrics grade a corrected sentence as a whole, and reference-free ones especially hide their reasoning. That opacity blocks model debugging and useful learner feedback.
Their method assigns each edit a Shapley value, borrowed from game theory, measuring its contribution to the sentence score. The attributions stayed consistent across edit granularities and matched human judgments about seventy percent of the time. Mining the attributions exposed metric biases, such as a tendency to ignore spelling-level fixes. The work turns a single number into an audit trail for grammatical error correction.
Catalog record
| CREATOR | Takumi Goto, Justin Vasselli, Taro Watanabe |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13110
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction Takumi Goto, Justin Vasselli, Taro Watanabe Abstract Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users. To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance. For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit. Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70% alignment with human evaluations. In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits. Our implementation is available at https://github.com/naist-nlp/gec-attribute . 1 Introduction Grammatical error correction (GEC) is the task of automatically correcting grammatical or superficial errors in an input sentence. Automatic evaluation metrics play a key role in improving GEC performance, but their effectiveness depends on their level of explainability. For example, metrics that evaluate at the edit level are more explainable than sentence-level metrics, as they allow us to identify which specific edits are effective and which are not, even when a GEC system makes multiple edits. Such explainable metrics enable researchers to analyze the strengths and weaknesses of GEC models, providing valuable insights into how models can be improved. Furthermore, in education applications, explainable metrics can provide language learners with detailed feedback on their writing, supporting their learning more effectively. (a) The existing metrics are low-explainability. (b) Our proposed method improves explainability. Figure 1: Overview of the proposed method with an example using three edits. Figure (a) shows the low-explainability of existing metrics that only estimate the sentence-level score, but Figure (b) shows that the edit-level attribution solves this issue. In GEC, explainable reference-based metrics, such as ERRANT Felice et al. (2016) ; Bryant et al. (2017) are limited because references cannot account for all valid corrections. Preparing test data with comprehensive references is often impractical, especially when targeting domains such as medical or academic writing that differ from existing datasets. To address this issue, reference-free metrics have been proposed to evaluate corrected sentences without relying on references Choshen and Abend (2018) ; Yoshimura et al. (2020) ; Islam and Magnani (2021) ; Maeda et al. (2022) . Although these reference-free metrics achieve high correlation with human evaluations, many are designed to assign scores at the sentence level, limiting their explainability on individual edits. This lack of granularity makes it difficult to analyze how specific edits contribute to the overall sentence score. For example, as shown in Figure 1 , a metric evaluates a corrected sentence created by applying the three edits. As shown in Figure 1 , the sentence-level metric assigns an overall score of 0.75, but it does not indicate whether all edits are valid, or if both valid and invalid edits have been applied. To improve the explainability of metrics with low or no explanation, we propose attributing sentence-level scores to individual edits as illustrated in Figure 1 . In the proposed method, the total contribution of all edits is calculated as the difference between the scores of the input sentence and the corrected sentence. This difference is then attributed to the individual edits. For example, in Figure 1 , a difference of -0.05 is distributed among three edits with contributions of 0.2, 0.1, and -0.35. The attribution results are intrepreted using the sign and magnitude of these scores: the sign indicates whether an edit is the valid or invalid, while the magnitude represents the degree of its influence on the final sentence-level score. We employ Shapley values Shapley et al. (1953) from cooperative game theory to fairly distribute the total score among the edits. By considering various combinations edits, Shapley values allow us to precisely attribute each edit’s contribution to the overall sentence score, offering insights into their individual impact. Unlike previous feature attribution methods Lundberg and Lee (2017) ; Sundararajan et al. (2017) , the proposed method is novel in attributing the difference between the input sentence and the corrected sentence. In the experiments, we apply the proposed method to two popular reference-free metrics, SOME Yoshimura et al. (2020) and IMPARA Maeda et al. (2022) , as well as a fluency metric based on GPT-2 Radford et al. (2019) perplexity. The results show that the proposed attribution method produces consistent scores across different granularities of edits and that edits with larger absolute attribution scores align more closely with human evaluations. We introduce Shapley sampling values Strumbelj and Kononenko (2010) to mitigate the time-complexity issues of calculating Shapley values. Additionally, we demonstrate that the proposed method can explain metric decisions at both the sentence and corpus levels, categorized by error types. These analyses reveal the types of edits that metrics give more weight to, as well as provide insights into the strengths and weaknesses of GEC systems. 2 Background Edits in GEC. The GEC task aims to correct grammatical errors in a source sentence S and output a corrected sentence H . The differences between S and H are often represented as N edits \boldsymbol{e}=\{e_{i}\}_{i=1}^{N} to enable evaluation Dahlmeier and Ng (2012) ; Bryant et al. (2017) ; Gong et al. (2022) ; Ye et al. (2023) , ensembling Tarnavskyi et al. (2022) , and post-processing Sorokin (2022) at the edit level. These edits can be automatically extracted using edit extraction methods Felice et al. (2016) ; Bryant et al. (2017) ; Belkebir and Habash (2021) ; Korre et al. (2021) ; Uz and Eryiğit (2023) . Each edit typically includes a word-level span in S and its corresponding correction, although it may also include an error type Bryant et al. (2017) . The error type categorizes each edit, indicating the part-of-speech or grammatical aspect it relates to, which helps to analyze the strengths and weaknesses of the GEC systems. Sentence-level Metrics. A sentence-level metric M computes the score of the corrected sentence given the source sentence, denoted as M(H|S)\in\mathbb{R} . The source sentence is used to assess meaning preservation, as GEC requires correcting errors while maintaining the original meaning of the source sentence. This formulation has been adopted by several reference-free metrics Yoshimura et al. (2020) ; Islam and Magnani (2021) ; Maeda et al. (2022) ; Kobayashi et al. (2024a) . Sentence-level metrics aim to rank GEC systems in alignment with humans judgments, as evidenced by the fact that the meta-evaluation is performed using the correlation between metric-generated rankings or scores and those of humans. However, these metrics are limited to sentence-level scoring and cannot explain how individual edits contribute to the final score. 3 Method Our attribution method assumes that the overall contribution of edits is the difference in scores before and after correction. We distribute the difference \Delta M(H|S)=M(H|S)-M(S|S) across each edit \boldsymbol{e}=\{e_{i}\}_{i=1}^{N} , where M(S|S) is the score of the source sentence treated as its own corrected sentence. The goal of our attribution method is to compute the contribution for each edit denoted as \{\phi_{i}(M)\in\mathbb{R}\}_{i=1}^{N}