Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators

Feng Gu Zongxia Li Carlos Rafael Colon · 2025

Read Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators by Feng Gu Zongxia Li Carlos Rafael Colon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06778.txt.

Introduction

Feng Gu and co-authors test large language models as annotators in a 2025 paper. Labeling events in news data matters for tracking markets and trends, but expert coding is slow and costly. Their test covers a full workflow: dropping irrelevant documents, merging reports of the same event, and labeling what remains.

Fully automated model labels beat older text methods but still fell short of human experts. The picture changed when models assisted the experts instead. Help with grouping events cut time and mental effort, and experts agreed more with model-extracted details than with fully automated labels. The authors favor models as assistants, not replacements, for careful annotation work.

Catalog record

CREATORFeng Gu Zongxia Li Carlos Rafael Colon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06778

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators Feng Gu Zongxia Li Carlos Rafael Colon Abstract Event annotation is important for identifying market changes, monitoring breaking news, and understanding sociological trends. Although expert annotators set the gold standards, human coding is expensive and inefficient. Unlike information extraction experiments that focus on single contexts, we evaluate a holistic workflow that removes irrelevant documents, merges documents about the same event, and annotates the events. Although llm -based automated annotations are better than traditional tf-idf -based methods or Event Set Curation, they are still not reliable annotators compared to human experts. However, adding llm s to assist experts for Event Set Curation can reduce the time and mental effort required for Variable Annotation. When using llm s to extract event variables to assist expert annotators, they agree more with the extracted variables than fully automated llm s for annotation. 1 Introduction Quality data is crucial for making informed decisions. Bloomberg 1 1 1 https://www.bloomberg.com/ , for example, automatically extracts top-quality data from trusted sources. Companies like Scale AI 2 2 2 https://scale.com/ collect, curate, and annotate data for ai models (OpenAI) and applications (self-driving cars, mapping, AR/VR, robotics). 3 3 3 https://openai.com/ While human annotators excel at reference resolution and aligning extracted information, machine coding can handle larger inputs, enhancing data collection efficiency and scalability Mason et al. 2012 ; King and Lowe 2003 . The Uppsala Conflict Data Project Georeferenced Event Data, one of the earliest event datasets relying on human labour, covers only fewer than 300,000 events in over twenty years Sundberg 2013 . In comparison, automated data processes like the gdelt Project collect trillions of datapoints The GDELT Project 2021 . Figure 1: Our workflow for annotating events data begins with preprocessing incoming media news. A Support Vector Machine identifies highly relevant documents for manual review. During Event Set Curation, human annotators create unique event sets. Finally, annotators code the domain-specific variables. We apply lm -based similarity indices and use lm -extracted variables to aid manual processing. Annotating event datasets presents significant challenges. First, statistical models often encounter challenges with text extraction and formatting when processing large corpora automatically Schrodt and Van Brackle 2013 . Issues can arise from unreliable source documents, incorrect translations, conflicting information, and inadequate filtering. Second, empirical evaluations yield inconsistent results with extremely low confidence intervals O’Brien 2013 . Third, unlike document-level annotation tasks, event annotation requires expertise in synthesizing information from multiple documents Gao et al. 2024 ; Li et al. 2024a , which often involves additional steps beyond labelling, extraction, or annotation. Contemporary event datasets typically combine quantitative methods with manual efforts. Algorithms like unsupervised clustering group documents Golub 2006 ; Hakim et al. 2014 ; Probierz et al. 2022 ; Bouras and Tsogkas 2013 ; Li et al. 2025a before human annotators refine the results and extract information Elliott 2018 ; Pierre and Jackson 2014 . We present a case study on ai automation for gtd 4 4 4 https://www.start.umd.edu/data-tools/GTD (Global Temporal Events Database), whose researchers maintain a similar open-source temporal event database used by various organizations. By integrating ai into the event annotation pipeline, we demonstrate that state-of-the-art nlp techniques can reduce human effort in traditional workflows. Specifically, we show that large language models ( llm s) are far from expert quality annotations, but they can serve as effective assistants for humans to extract event variables, thus reducing the variable coding time of experts. More broadly, we outline the potential of integrating llm s into event annotation pipelines. Capturing semantic similarity and coding diverse variables with high expert agreement, lm s show strong promise for real-world annotation tasks. 2 Related Work Single-document Event Extraction: Prior work on information extraction focuses primarily on sentence and document-level tasks like part-of-speech tagging Roth and Zelenko 1998 and named-entity recognition Collobert et al. 2011 , creating various benchmarks. Gao et al. 2024 introduces cross-document event extraction, which involves merging duplicates and resolving conflicts. Our task aligns with cross-document event extraction but is more complex: an lm must identify documents about the same event and extract information adhering to a strictly defined schema. Variable Extraction as a qa Task: Neural network models excel in qa benchmarks Hermann et al. 2015 ; Li et al. 2025b . Extracting domain-specific variables can be framed as a qa task, leveraging contextual information from coding criteria. However, evaluating qa systems is challenging in contextual or complex output spaces Xu et al. 2023 ; Bulian et al. 2022 , often lacking ecological validity. We use trained rule-based and neural matching methods to achieve a more robust variable equivalence detection. Text Segmentation: Text segmentation divides large text blocks into smaller, topically coherent sections. Koshorek et al. 2018 create an annotated corpus using Wikipedia pages, while Aumiller et al. 2021 use a bert -based model for legal documents. Neural models have achieved excellent performance Glavaš and Somasundaran 2020 . Although algorithms like TextTiling Hearst 1997 and TopicTiling Riedl and Biemann 2012 can handle texts with distinct sections, documents with closely related events like ours pose greater segmentation challenges. llm -assisted Annotation: Annotating domain-specific data requires expertise. Recent nlp advancements enable llm s to annotate data efficiently, achieving high accuracy on document-level datasets Tan et al. 2024 . However, Tseng et al. 2024 show lm s cannot replace experts in financial data annotation. Similarly, Yadav et al. 2024 finds gpt -4 performs poorly in annotating word semantics. Our work explores llm -assisted annotation in a real-world even annotation workflow. 3 A Case Study of Event Annotation The existing data workflow at the gtd consists of automated and manual processing (Figure 1 ). To obtain quality data from news articles that can be later used for analysis, experts 1) compile a list of unique events from a collection of documents and 2) extract synthesized event information. 3.1 Event Set Curation An event set is a collection of documents about the same event. Unlike document-level annotations, event details are scattered across multiple documents, each offering partial or complementary information. Annotators must prioritize informative documents (e.g., those with high tf-idf scores), systematically review them to identify unique events, and create event sets that meet predefined criteria and sending them for downstream coding. For example, one document describes an incident “ in San Carlos City on Sunday, where a men attack a vehicle in Barangay Palampas, ” while another specifies that “ the incident occurred in Barangay Palampas, San Carlos City, Negros Occidental, on February 20, ” providing precise details about the incident’s date. To ensure consistency in event sets for downstream comparisons, we evaluate candidate event sets using precision, recall, and F_{1} score. 3.2 Variable Coding Annotators synthesize information from documents in event set for each event, identifying variables like location and perpetrators 5 5 5 Full list and description in Appendix A.5 . . One document describes an incident involving “ approximately 20 people, some with axes, who caused injuries ” in “ Houston, British Columbia, ” while another document states that “ far-left anti-pipeline extremists ” attacked the “ Morice River drill pad site off the Marten Forest Service Road. ” Annotators use information from all relevant documents to code variables.

Read the complete text