ConfliBERT: A Language Model for Political Conflict

Patrick T. Brandt; UT Dallas, [email protected] · 2024

Read ConfliBERT: A Language Model for Political Conflict by Patrick T. Brandt; UT Dallas, [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15060.txt.

Introduction

A research team published a research paper in 2024 on ConfliBERT, a language model specialized for political conflict and violence. Scholars long scraped news with rigid rules to log who did what to whom. A model tuned on conflict text can read more flexibly. The paper reviews the model and its uses for classifying actors and actions in such reports.

Fine-tuned, it beat big generalist models from Google, Meta, and Alibaba on accuracy, precision, and recall within its domain, while running hundreds of times faster. Examples draw on BBC texts and two established conflict data sets. The authors argue small specialist models still outgun giants where the vocabulary and stakes are specific.

Catalog record

CREATORPatrick T. Brandt; UT Dallas, [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-15060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ConfliBERT: A Language Model for Political Conflict Patrick T. Brandt; UT Dallas, [email protected] Abstract Conflict scholars have used rule-based approaches to extract information about political violence from news reports and texts. Recent Natural Language Processing developments move beyond rigid rule-based approaches. We review our recent ConfliBERT language model (Hu et al. 2022) to process political and violence related texts. The model can be used to extract actor and action classifications from texts about political conflict. When fine-tuned, results show that ConfliBERT has superior performance in accuracy, precision and recall over other large language models (LLM) like Google’s Gemma 2 (9B), Meta’s Llama 3.1 (7B), and Alibaba’s Qwen 2.5 (14B) within its relevant domains. It is also hundreds of times faster than these more generalist LLMs. These results are illustrated using textsfrom the BBC, re3d, and the Global Terrorism Dataset (GTD). Previous versions have been presented at the virtual Methods in Event Detection Colloquium (September 2024). This research was supported by NSF award 2311142; used Delta at NCSA / University of Illinois through allocation CIS220162 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF grants 2138259, 2138286, 2138307, 2137603, and 2138296. This material is based upon High Performance Computing (HPC) resources supported by the University of Arizona TRIF, UITS, and Research, Innovation, and Impact (RII). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, NCSA, or the affiliated university resource providers. Thanks for suggestions go to Scott Althaus, R. Michael Alvarez, Ben Bagozzi, Rebecca Cordell, Ryan Kennedy, Hyein Kim, Shahryar Minhas, Philip Schrodt, and Nora Webb Williams. 1 Introduction A common text as data problem in the social sciences is the need to migrate information in a large corpora into a data summary of that text. As social (political) scientists we possess replicable and encoded domain expertise to understand these texts. There are components of the processing and summarization of large corpora that are better understood by experts in machine learning and natural language processing (NLP). How then should one combine the insights of domain experts and computational and information scientists of natural language? Here we present one possible answer using recent large language models to illustrate how domain expertise and computational science can be combined. The conflict event data we care about are typically from human-coded news reports. The transformation of unstructured news texts into structured "who-did-what-to-whom" event data is fundamental in the international relations, and studies of conflict and political violence. The process of gathering and preparing data for analysis in this domain is: assemble a corpus, filter for relevant information, identify target events, and annotate event attributes. This process can be costly and time consuming: time required for data collection, filtering the large amount of unstructured text, training human annotators on the usage of the applied ontology, and several rounds of quality controls to ultimately achieve a curated text corpus. This parallels the widely systematized way to approach text-as-data in much of international relations and the social sciences ( O’Connor et al.,, 2013 ; Grimmer et al.,, 2022 ) . Ultimately, these pre-processing steps are applied to the unstructured corpus to permit the extraction of relevant information on actors (who), actions (did what) and the targets (to whom) for political science research. This transformation of unstructured news text faces several well-known challenges that researchers in the field have long faced. 1 1 1 We presume the existence of a relevant corpus of texts in digital form, akin to economists’ can-opener assumption. These can be categorized into three main areas: 1. Filtering for relevant information or reports 2. Identifying events 3. Annotating event attributes While these areas can be challenging, they align with common tasks in language learning and computational information science. In NLP these latter tasks are addressed through techniques such as semantic role learning (SRL), automated content extraction (ACE). Here we address how these problems can be addressed with a language model called ConfliBERT or a Bidirectional Encoder Representations from Transformers (BERT) model trained on conflict data. ConfliBERT, as a political ‘‘conflict’’ informed model, does this using a BERT model. 2 2 2 The conflict domain part is first and the LLM / AI tool is second. Since we started this in 2021, the LLM tool of the time was BERT. However, we further introduce a conflict-specific adaptation of LLAMA ( Dubey et al.,, 2024 ) , a generative AI-based model, which we termed ConfLlama and trained on conflict-specific training data in section 4 to permit a performance comparison between ConfliBERT and current generative AI models. ConfliBERT is a language model trained on domain-specific conflict and political violence data and achieves superior performance in the summarization and classification tasks political scientists perform on unstructured text corpora. ConfliBERT (for English) is publicly available at Hugging Face, and is trained on an expert-curated corpus of 33.7 GB about conflict and political violence. Combining domain-specific knowledge and computational tools in ConfliBERT improves unstructured conflict texts processing. The method discussed here is a domain-specific pretrained language model implemented for English ( Hu et al.,, 2022 ) , and there are variants for Spanish ( Yang et al.,, 2023 ) , and Arabic ( Alsarra et al.,, 2023 ) . ConfliBERT, as a political science trained model, can help encode information about events from text without having to have a fully specified ontology of actors or their interactions (a major impediment in earlier domain-based methods). Our proposed method uses a transformer language models ( Devlin et al.,, 2018 ) and large amounts of politically relevant news texts as training data. We re-cast the problems of the political science domain into those more commonly seen in the information and computer science domains of text and natural language processing and inferences. This trades human annotation and classification costs for computational resources (which grow more powerful and cheaper), while augmenting the speed of data generation. However, how this is done in the computational linguistics and information sciences community needs to be bridged to how social scientists think about these problems. There the focus is on annotation of spans of text that correspond to linguistic and contextual entities (by human or machine). In contrast, political and social scientists focus on event attributes, their modality and characteristics ( Olsen et al.,, 2024 ) . Similar domain-specific BERT models have shown to equally outperform generic BERT models in text classifications in other scientific fields such as: biomedical SCIBERT ( Beltagy et al.,, 2019 ) , material sciences, MatSCIBERT ( Gupta et al.,, 2022 ) , legal LegalBERT Chalkidis et al., (2020) , finance (FinBERT) Araci, (2019) , clinical notes (ClinicalBERT) ( Huang et al.,, 2020 ) , and patent texts (patentBERT) ( Lee and Hsiang,, 2019 ) . For the political science domain we want to use ConfliBERT to accomplish three key information extraction and summarization tasks that are part of "coding event data": Filtering for Relevant Information : In political science, the sheer volume of available text data — from news articles and social media posts to diplomatic cables and policy documents — necessitates efficient filtering. The challenge lies not just

Read the complete text