The CoNLL-2013 Shared Task on Grammatical Error Correction

Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, Joel Tetreault · 2025

Read The CoNLL-2013 Shared Task on Grammatical Error Correction by Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, Joel Tetreault (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09474.txt.

Introduction

Ng and co-authors describe a shared contest on fixing grammar errors in learner English, in this paper on the CoNLL-2013 task. Such contests give research teams the same data and scoring, so methods can be compared fairly. Grammar correction later grew into tools used by writers everywhere.

The paper defines the task, presents the data sets, and explains the metric and scorer. It then surveys the approaches the teams brought, from rule systems to statistical methods of the day. Final rankings show which ideas worked best on unseen essays. The report remains a reference point for the error correction field.

Catalog record

CREATORHwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, Joel Tetreault
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The CoNLL-2013 Shared Task on Grammatical Error Correction Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, Joel Tetreault Abstract The CoNLL-2013 shared task was devoted to grammatical error correction. In this paper, we give the task definition, present the data sets, and describe the evaluation metric and scorer used in the shared task. We also give an overview of the various approaches adopted by the participating teams, and present the evaluation results. 1 Introduction Grammatical error correction is the shared task of the Seventeenth Conference on Computational Natural Language Learning in 2013 (CoNLL-2013). In this task, given an English essay written by a learner of English as a second language, the goal is to detect and correct the grammatical errors present in the essay, and return the corrected essay. This task has attracted much recent research interest, with two shared tasks Helping Our Own (HOO) 2011 and 2012 organized in the past two years [ \citenameDale and Kilgarriff2011 , \citenameDale et al.2012 ] . In contrast to previous CoNLL shared tasks which focused on particular subtasks of natural language processing, such as named entity recognition, semantic role labeling, dependency parsing, or coreference resolution, grammatical error correction aims at building a complete end-to-end application. This task is challenging since for many error types, current grammatical error correction systems do not achieve high performance and much research is still needed. Also, tackling this task has far-reaching impact, since it is estimated that hundreds of millions of people worldwide are learning English and they benefit directly from an automated grammar checker. The CoNLL-2013 shared task provides a forum for participating teams to work on the same grammatical error correction task, with evaluation on the same blind test set using the same evaluation metric and scorer. This overview paper contains a detailed description of the shared task, and is organized as follows. Section 2 provides the task definition. Section 3 describes the annotated training data provided and the blind test data. Section 4 describes the evaluation metric and the scorer. Section 5 lists the participating teams and outlines the approaches to grammatical error correction used by the teams. Section 6 presents the results of the shared task. Section 7 concludes the paper. 2 Task Definition The goal of the CoNLL-2013 shared task is to evaluate algorithms and systems for automatically detecting and correcting grammatical errors present in English essays written by second language learners of English. Each participating team is given training data manually annotated with corrections of grammatical errors. The test data consists of new, blind test essays. Preprocessed test essays, which have been sentence-segmented and tokenized, are also made available to the participating teams. Each team is to submit its system output consisting of the automatically corrected essays, in sentence-segmented and tokenized form. Grammatical errors consist of many different types, including articles or determiners, prepositions, noun form, verb form, subject-verb agreement, pronouns, word choice, sentence structure, punctuation, capitalization, etc. Of all the error types, determiners and prepositions are among the most frequent errors made by learners of English. Not surprisingly, much published research on grammatical error correction focuses on article and preposition errors [ \citenameHan et al.2006 , \citenameGamon2010 , \citenameRozovskaya and Roth2010 , \citenameTetreault et al.2010 , \citenameDahlmeier and Ng2011b ] , with relatively less work on correcting word choice errors [ \citenameDahlmeier and Ng2011a ] . Article and preposition errors were also the only error types featured in the HOO 2012 shared task. Likewise, although all error types were included in the HOO 2011 shared task, almost all participating teams dealt with article and preposition errors only (besides spelling and punctuation errors). In the CoNLL-2013 shared task, it was felt that the community is now ready to deal with more error types, including noun number, verb form, and subject-verb agreement, besides articles/determiners and prepositions. Table 1 shows examples of the five error types in our shared task. Error tag Error type Example sentence Correction (edit) ArtOrDet Article or determiner In late nineteenth century, there was a severe air crash happening at Miami international airport. late \rightarrow the late Prep Preposition Also tracking people is very dangerous if it has been controlled by bad men in a not good purpose. in \rightarrow for Nn Noun number I think such powerful device shall not be made easily available. device \rightarrow devices Vform Verb form However, it is an achievement as it is an indication that our society is progressed well and people are living in better conditions. progressed \rightarrow progressing SVA Subject-verb agreement People still prefers to bear the risk and allow their pets to have maximum freedom. prefers \rightarrow prefer Table 1: The five error types in our shared task. Since there are five error types in our shared task compared to two in HOO 2012, there is a greater chance of encountering multiple, interacting errors in a sentence in our shared task. This increases the complexity of our shared task relative to that of HOO 2012. To illustrate, consider the following sentence: Although we have to admit some bad effect which is brought by the new technology, still the advantages of the new technologies cannot be simply discarded. The noun number error effect needs to be corrected (effect \rightarrow effects). This necessitates the correction of a subject-verb agreement error (is \rightarrow are). A pipeline system in which corrections for subject-verb agreement errors occur strictly before corrections for noun number errors would not be able to arrive at a fully corrected sentence for this example. The ability to correct multiple, interacting errors is thus necessary in our shared task. The recent work of [ \citenameDahlmeier and Ng2012a ] , for example, is designed to deal with multiple, interacting errors. Note that the essays in the training data and the test essays naturally contain grammatical errors of all types, beyond the five error types focused in our shared task. In the automatically corrected essays returned by a participating system, only corrections necessary to correct errors of the five types are made. The other errors are to be left uncorrected. 3 Data This section describes the training and test data released to each participating team in our shared task. 3.1 Training Data The training data provided in our shared task is the NUCLE corpus, the NUS Corpus of Learner English [ \citenameDahlmeier et al.2013 ] . As noted by [ \citenameLeacock et al.2010 ] , the lack of a manually annotated and corrected corpus of English learner texts has been an impediment to progress in grammatical error correction, since it prevents comparative evaluations on a common benchmark test data set. NUCLE was created precisely to fill this void. It is a collection of 1,414 essays written by students at the National University of Singapore (NUS) who are non-native speakers of English. The essays were written in response to some prompts, and they cover a wide range of topics, such as environmental pollution, health care, etc. The grammatical errors in these essays have been hand-corrected by professional English instructors at NUS. For each grammatical error instance, the start and end character offsets of the erroneous text span are marked, and the error type and the correction string are provided. Manual annotation is

Read the complete text