Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data

Poli Apollinaire Nemkova, Solomon Ubani, Mark V. Albert · 2025

Read Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data by Poli Apollinaire Nemkova, Solomon Ubani, Mark V. Albert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10260.txt.

Introduction

Nemkova, Ubani, and Albert test how well large language models label social media posts in a 2025 study. The posts, in Russian and Ukrainian, had to be marked for whether they refer to human rights abuses. That task is hard because the calls are often matters of context and judgment, not just key words.

The team compared five models, among them GPT-4 and Claude-2, against labels made twice by human readers for 1,000 posts. They tried prompts in English and Russian, with no examples and with a few. Results show where each model agrees with human readers, where it fails, and how prompt language changes its work, a guide for use on sensitive tasks.

Catalog record

CREATORPoli Apollinaire Nemkova, Solomon Ubani, Mark V. Albert
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data Poli Apollinaire Nemkova, Solomon Ubani, Mark V. Albert Abstract Large language models (LLMs) have shown promise in tasks requiring nuanced textual understanding. This study evaluates state-of-the-art LLMs—GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2—for annotating a multilingual dataset of social media posts in Russian and Ukrainian, focusing on identifying human rights violations. Model performance is compared against a human-annotated gold standard across 1000 samples, with analysis of zero-shot and few-shot prompting in English and Russian. Key findings show GPT-4.0 excels in precision and recall, making it ideal for high-stakes tasks. Aligning prompt language with dataset language improves performance, particularly for open-source models like LLaMA3 and Mistral-7B, which benefit from few-shot prompting but require additional validation for sensitive applications. Models, like humans, tend to struggle with ambiguous cases, particularly those involving indirect or nuanced references. Closed-source models demonstrate robust zero-shot capabilities with marginal gains from few-shot setups. This study highlights the importance of linguistic alignment, prompt strategy, and model selection, offering practical insights for deploying LLMs in multilingual, high-impact domains like human rights monitoring. 1 Introduction The emergence of large language models (LLMs) has revolutionized natural language processing (NLP), pushing the boundaries of what machines can achieve in understanding, generating, and classifying text. Models such as GPT-3.5, GPT-4, Claude-2, and open-source alternatives like LLaMA and Mistral-7B have demonstrated extraordinary capabilities across a wide range of applications, from translation to content summarization. However, the deployment of LLMs for high-stakes, domain-specific tasks, such as identifying references to human rights violations in multilingual social media data, remains a critical yet underexplored area of research. The domain of human rights monitoring presents unique challenges. It involves the analysis of contextually rich, multilingual, and emotionally charged texts, often requiring nuanced comprehension and interpretation of implicit or indirect references. Traditionally, these tasks have been conducted by human annotators, whose expertise ensures high accuracy but incurs significant costs in time and resources. LLMs offer an unprecedented opportunity to automate and scale such tasks, but their ability to handle the inherent subjectivity and complexity of human rights-related data is not well understood. Moreover, questions remain about how factors such as prompt design, linguistic alignment, and model architecture influence performance. In this study, we systematically evaluate the performance of leading LLMs and open-source alternatives on the binary classification task of detecting references to human rights violations in social media posts written in Russian and Ukrainian. Using a dataset curated from public Telegram channels, we compare models under two prompting configurations—zero-shot and few-shot learning—and investigate the effect of prompt language (English vs. Russian) on model performance. Additionally, we examine differences between closed-source models, such as GPT-4 and Claude-2, and open-source models, such as LLaMA and Mistral-7B, to understand the trade-offs between cost, accessibility, and performance. Key contributions of this study: • Comprehensive Benchmarking Across Models and Prompts: We provide a comparative analysis of closed- and open-source LLMs under different prompting styles (zero-shot vs. few-shot) and languages (English vs. Russian), highlighting performance gaps and strengths. • Insights Into Prompting Strategies: The study demonstrates that open-source models benefit significantly from few-shot prompting with examples, whereas closed-source models exhibit robust performance across both zero-shot and few-shot settings. • Impact of Linguistic Alignment: We show that aligning the prompt language with the data language (e.g., Russian prompts for Russian texts) consistently improves model performance across architectures, emphasizing the importance of multilingual capability in real-world tasks. • Error Analysis and Practical Guidance: Through a detailed analysis of error types and model disagreements with human annotations, we provide actionable insights for selecting and optimizing LLMs in high-context, sensitive domains. • Cost-Effectiveness of Open-Source Models: Despite their lower baseline performance, open-source models demonstrate strong potential when optimized with appropriate prompting strategies, making them viable alternatives for resource-constrained scenarios. This work advances the understanding of LLM capabilities in multilingual, high-stakes applications and offers practical guidelines for leveraging these models effectively in real-world human rights monitoring tasks. By addressing critical gaps in model evaluation and exploring the interplay of architecture, prompt design, and language alignment, this study lays the groundwork for future research and development in this domain. 2 Literature Review The emergence of Large Language Models (LLMs) has transformed Natural Language Processing (NLP), particularly in text annotation, enabling automation at a scale and speed unattainable through traditional methods. This review explores the comparative effectiveness of LLMs and human annotators in text annotation tasks ( Aldeen et al. 2023 ; Nasution and Onan 2024 ) , highlighting their strengths, limitations, and the pressing need for further research into their labeling performance, especially for complex and high-context tasks. Efficiency and Cost-Effectiveness. LLMs have demonstrated substantial potential to reduce the time and cost associated with text annotation. For instance, integrating LLMs with human expertise in medical information extraction has been shown to significantly reduce manual effort while maintaining high accuracy, facilitating the rapid creation of labeled datasets ( Goel et al. 2023 ) . Similarly, the CoAnnotating framework leverages LLMs to complement human annotators by allocating annotation tasks based on uncertainty, achieving a 21% performance improvement over random baselines ( Li et al. 2023 ) . However, these gains are task-dependent. For more nuanced annotations, such as legal or ethical contexts, the costs of fine-tuning or prompt engineering can be substantial, offsetting the initial efficiency benefits ( Brown and Others 2020 ; Bommasani et al. 2021 ) . This highlights a critical need to understand where LLMs can outperform or meaningfully complement traditional annotation workflows. Accuracy and Reliability. While LLMs excel in generalization, particularly in zero-shot and few-shot learning, their performance varies significantly across tasks and datasets. For example, ChatGPT-4 has outperformed both expert classifiers and crowd workers in annotating political Twitter messages, demonstrating higher accuracy and consistency ( Törnberg 2023 ) . Similarly, GPT-3 and GPT-4 have shown strong performance in summarizing legal documents and medical records, often rivaling domain experts ( Šavelka 2023 ; Shaib et al. 2023 ; Takagi et al. 2023 ) . However, studies have also highlighted limitations. For instance, LLMs frequently underperform on tasks requiring deep contextual understanding or domain-specific expertise, often lagging behind fine-tuned smaller models trained on expert-annotated data ( Plaza-del Arco, Nozza, and Hovy 2023 ; Pangakis, Wolken, and Fasching 2023 ) . Furthermore, while LLMs achieve high accuracy in standard tasks, their susceptibility to generating false positives or negatives in edge cases or nuanced scenarios remains a critical limitation. Domain-Specific Knowledge. Human annotators possess domain-specific knowledge that is often critical for accurate text annotation. In tasks requiring subtle contextual judgments, such as medical diagnosis or human rights monitoring, small models trained on curated datasets have been shown to outperform general-purpose LLMs ( Lu et al. 2023 ) . These findings underscore the limitations of pre-trained LLMs when applied to tasks requiring specialized knowledge and suggest that domain adaptation remains a key challenge. Research also indicates that while in-context learning (e.g., few-shot prompting) improves LLM performance in specialized domains, it is not always sufficient to match the nuanced understanding of human annotators ( Shin et al. 2020 ) . This highlights a need for further exploration into how prompting strategies can bridge this gap, particularly in complex labeling tasks.

Read the complete text