Build, Borrow, or Just Fine-Tune? A Political Scientist’s Guide to Choosing NLP Models
Shreyas Meher · 2026
Read Build, Borrow, or Just Fine-Tune? A Political Scientist’s Guide to Choosing NLP Models by Shreyas Meher (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09595.txt.
Introduction
Shreyas Meher guides political scientists through a practical choice in natural language processing: build a domain model, borrow and adapt one, or fine-tune a general model. Using conflict event classification as the test case, Meher fine-tunes ModernBERT on the Global Terrorism Database to create Confli-mBERT. It trails the domain-pretrained ConfliBERT by about four accuracy points overall. The gap concentrates in rare event categories, while on frequent attack types the two models are nearly indistinguishable.
Catalog record
| CREATOR | Shreyas Meher |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71206 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09595
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Build, Borrow, or Just Fine-Tune? A Political Scientist’s Guide to Choosing NLP Models Shreyas Meher Abstract Political scientists increasingly face a consequential choice when adopting natural language processing tools: build a domain-specific model from scratch, borrow and adapt an existing one, or simply fine-tune a general-purpose model on task data? Each approach occupies a different point on the spectrum of performance, cost, and required expertise, yet the discipline has offered little empirical guidance on how to navigate this trade-off. This paper provides such guidance. Using conflict event classification as a test case, I fine-tune ModernBERT on the Global Terrorism Database (GTD) to create Confli-mBERT and systematically compare it against ConfliBERT, a domain-specific pretrained model that represents the current gold standard. Confli-mBERT achieves 75.46% accuracy compared to ConfliBERT’s 79.34%. Critically, the four-percentage-point gap is not uniform: on high-frequency attack types such as Bombing/Explosion (F1 = 0.95 vs. 0.96) and Kidnapping (F1 = 0.92 vs. 0.91), the models are nearly indistinguishable. Performance differences concentrate in rare event categories comprising fewer than 2% of all incidents. I use these findings to develop a practical decision framework for political scientists considering any NLP-assisted research task: when does the research question demand a specialized model, and when does an accessible fine-tuned alternative suffice? The answer, I argue, depends not on which model is “better” in the abstract, but on the specific intersection of class prevalence, error tolerance, and available resources. The model, training code, and data are publicly available on Hugging Face. Keywords natural language processing \cdot model selection \cdot political methodology \cdot conflict event classification \cdot transformer models \cdot fine-tuning \cdot Global Terrorism Database 1 Introduction Political scientists are adopting artificial intelligence at an unprecedented rate. Every year brings new applications: automated coding of conflict events ( 6 ) , sentiment analysis of legislative debate ( 17 ) , classification of political claims and media content ( 7 ) , and large-scale annotation of survey responses ( 4 ) . The transformer revolution ( 19 ; 3 ) has made these applications not only possible but increasingly accessible, lowering the technical barrier so far that a researcher with a laptop and a weekend can build a working text classifier. Yet this very accessibility has created a new kind of methodological puzzle. When a political scientist sets out to classify text for a research project, she now faces a bewildering landscape of options. At one extreme, she can use a general-purpose model like BERT or GPT straight off the shelf. At the other, she can invest weeks or months building a domain-specific model pretrained on a curated corpus of political texts. In between lies a growing middle ground of fine-tuning strategies: taking a general-purpose model and adapting it to a specific task using labeled data. The academic literature has, understandably, focused on pushing the frontier. Domain-specific models such as ConfliBERT ( 1 ) , LegalBERT ( 2 ) , and BioBERT ( 11 ) have demonstrated that specialized pretraining can improve downstream performance. These are genuine scientific contributions that have advanced their respective fields. But the performance comparisons in these papers implicitly frame the choice as one-dimensional: more investment yields better performance, and better performance is always worth pursuing. What is missing from this conversation is a sober assessment of the trade-offs involved. Building a domain-specific model is expensive, technically demanding, and requires ongoing maintenance as architectures evolve. Fine-tuning a general-purpose model is cheap, fast, and reproducible. The question that most practitioners actually face is not “which approach is theoretically superior?” but “which approach is appropriate for my research question, given my resources?” This is a question about matching tools to tasks, and it is the question this paper addresses. I approach it empirically. Using conflict event classification on the Global Terrorism Database as a test case, I compare ConfliBERT, a domain-specific pretrained model that represents the gold standard for this task, against Confli-mBERT, a model I construct by simply fine-tuning ModernBERT ( 20 ) , a freely available general-purpose transformer, on the same labeled data. The comparison is deliberately designed to be as informative as possible about the marginal value of domain-specific pretraining : both models are evaluated on identical data, with identical metrics, on the same nine-category classification task. The headline result is a four-percentage-point accuracy gap (75.46% vs. 79.34%) in ConfliBERT’s favor. But the headline result is not the interesting part. What is interesting is where that gap comes from. On the attack types that dominate the empirical literature and that together account for over 98% of observed incidents, the two models perform comparably. The gap concentrates almost entirely in rare event categories (Hijacking, Barricade Incidents, Unarmed Assault) where both models struggle and where any automated approach would benefit from human verification. From these findings, I develop a decision framework that generalizes beyond conflict studies. The framework identifies three factors that should guide a political scientist’s choice of NLP model: the prevalence distribution of the categories being classified, the error tolerance of the downstream analysis, and the available computational and human resources. I argue that for the majority of political science applications, fine-tuning provides a practical, accessible, and scientifically defensible starting point, and that domain-specific models like ConfliBERT represent a valuable next step for researchers whose specific research questions demand higher performance on rare or specialized categories. The contribution of this paper is therefore not a new model per se, but a new way of thinking about an increasingly common methodological decision. As AI tools become standard equipment in the discipline, political scientists need principled guidance on which tools to use when. This paper offers that guidance, grounded in an empirical comparison that makes the trade-offs concrete. The remainder of the paper is organized as follows. Section 2 positions the model selection problem within the broader evolution of NLP methods in political science. Section 3 describes the GTD data used as the empirical test case. Section 4 details the fine-tuning methodology. Section 5 presents the comparison between Confli-mBERT and ConfliBERT. Section 6 develops the decision framework and discusses its implications. Section 7 concludes with practical recommendations. 2 Related Work 2.1 The Build-Borrow-Buy Spectrum in NLP for Political Science Political scientists who wish to apply NLP to their research have historically faced three broad options, each representing a different level of investment and customization. The first option, which we might call buying , involves using a pre-existing model or API with minimal adaptation. A researcher might submit texts to a general-purpose large language model through an API, or use a pre-trained classifier with zero-shot or few-shot prompting. This approach requires the least technical expertise, but it offers no guarantee that the model’s internal representations are well-suited to the political science domain. Work by 7 has explored this frontier, examining how off-the-shelf large language models perform on political text coding tasks. The second option, building , involves pretraining a model from scratch on domain-specific data. This is the approach taken by ConfliBERT ( 1 ) for conflict studies, LegalBERT ( 2 ) for legal NLP, and BioBERT ( 11 ) for biomedical text mining. Building yields the most customized model, potentially with the best domain-specific performance, but it requires assembling a large curated corpus,