Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy

AJ Alvero, Ruohong Dong, Klint Kanopka, David Lang · 2025

Read Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy by AJ Alvero, Ruohong Dong, Klint Kanopka, David Lang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08199.txt.

Introduction

Sociologists increasingly analyze text with computer science tools, and the newest tool is usually assumed to be the best. AJ Alvero and 3 co-authors challenge that assumption in a 2025 research paper. They call it the state-of-the-art fallacy. Methods rise in computer science by beating accuracy benchmarks, which may not match what a sociological question needs.

The authors revisit a study of college admissions essays. They rerun the analysis with older and newer methods side by side. The newer techniques did not meaningfully beat the earlier results. Generative AI and large language models also introduced bias and confounding that were hard to remove. The authors argue for methodological pluralism: choose tools that fit the theory and the question. The warning applies well beyond sociology, wherever benchmark prestige drives method choice.

Catalog record

CREATORAJ Alvero, Ruohong Dong, Klint Kanopka, David Lang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08199

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Algorithmic Tradeoffs, Applied NLP, and the State-of-the-Art Fallacy AJ Alvero, Ruohong Dong, Klint Kanopka, David Lang Abstract Computational sociology is growing in popularity, yet the analytic tools employed differ widely in power, transparency, and interpretability. In computer science, methods gain popularity after surpassing benchmarks of predictive accuracy, becoming the “state of the art.” Computer scientists favor novelty and innovation for different reasons, but prioritizing technical prestige over methodological fit could unintentionally limit the scope of sociological inquiry. To illustrate, we focus on computational text analysis and revisit a prior study of college admissions essays, comparing analyses with both older and newer methods. These methods vary in flexibility and opacity, allowing us to compare performance across distinct methodological regimes. We find that newer techniques did not outperform prior results in meaningful ways. We also find that using the current state of the art, generative AI and large language models, could introduce bias and confounding that is difficult to extricate. We therefore argue that sociological inquiry benefits from methodological pluralism that aligns analytic choices with theoretical and empirical questions. While we frame this sociologically, scholars in other disciplines may confront what we call the “state-of-the-art fallacy”, the belief that the tool computer scientists deem to be the best will work across topics, domains, and questions. K eywords college admissions, sociological imagination, computational sociology, natural language processing, machine learning Introduction In computational research, whether sociologically aligned or otherwise, analysts inevitably encounter a critical question: which modeling approach should I/we take? Model selection is a multifaceted process that requires weighing a variety of tradeoffs. In the machine learning literature, the well-known bias-variance tradeoff encapsulates the tension between statistical bias and variance hastie2009elements ; james2013introduction . From a more theoretical perspective, scholars emphasize the importance of balancing flexibility, accuracy, and parsimony in model selection as a way to ensure strong results that are still inferable xie1998comment . In computational text analysis, a topic gaining prominence in sociology evans2016machine , these tradeoffs take on unique forms. Modeling decisions in this area often reflect not only the balance between interpretability and flexibility but also entirely distinct approaches to representing text and generating variables. For instance, topic modeling generates representations of documents as mixtures of word co-occurrence clusters referred to as “topics.” In contrast, word embeddings focus on the relationships of individual words within a corpus, eschewing document-level representations in favor of high-dimensional word spaces. Beyond these two popular sets of methods, there are entire conferences and sub-fields dedicated to developing tools primarily geared for text analysis, making it difficult if not impossible to choose the one truly “right” method. Therefore, as our social worlds become increasingly text-saturated and tools to analyze such data improve and popularize, scholars face the challenge of determining which computational methods best align with their research questions. Sociologists who advocate machine learning frameworks have articulated compelling arguments for the adoption of computational approaches as a means to leverage our modern data deluge borch_oxford_nodate ; rahal2024rise . For example, Verhagen verhagen2024incorporating highlights how machine learning methods can enhance hypothesis testing and refine applied models. Similarly, Lundberg et al. lundberg2022researcher discuss the potential of computational techniques to improve prediction and analysis in large datasets, offering insight into social phenomena that have the potential to surpass traditional statistical methods. However, these contributions often focus on general applications of computational techniques rather than on their specific implications for text analysis. Although sociologists have contributed to the development of computational tools adapted for social science, many of the widely adopted methods in machine learning and natural language processing (NLP) are developed by computer scientists whose priorities differ from those of social scientists. These tools are typically optimized for achieving maximum predictive accuracy on benchmark datasets rather than addressing the nuanced demands of social inquiry luitse2024ai ; koch2024protoscienceepistemicmonoculturebenchmarking . As Koch & Peterson note, not a bottom-up cultural development in computer science but rather a heavily top-down phenomenon influenced by major funding sources like DARPA over decades. A result of this arrangement is that distinctions between algorithms that model text as mixtures of topics versus those that represent words in high-dimensional spaces or any other approach may seem less relevant in computer science than in sociology. Modeling decisions for computer scientists tend to be based on “whatever scored highest on the latest benchmark” donoho201750 ; koch2021reducedreusedrecycledlife rather than ‘‘whatever makes the most sense for the particular research question.’’ These dynamics can limit computational social scientific research as well by pushing analysts to adopt the state of the art 1 1 1 Typically, “state of the art” is a noun and “state-of-the-art” is an adjective, a convention we follow in this article. method as determined by computer science in all situations or even punish authors who use older methodological approaches. These dynamics also point to a potential “file drawer problem” rosenthal1979file where computational and/or social scientists might be limiting the breadth of their approach in favor of whatever has become or currently is popular. Identifying this trend in the literature would therefore be difficult, though in this paper we take more of a conceptual approach and compare analytical frameworks comparing older methods with newer ones under the assumption some might have that the state of the art will generally work the best. Sociologists adapting text analysis methods obviously implies that sociologists are receiving some kind of training in these techniques, be it through formal coursework or through semi-formal means like workshops and focused trainings (e.g., ICPSR). The majority of the top sociology programs in the U.S. offer some kind of graduate focused course in computational methods and analysis (21 of the top 31; see Table A1 for a breakdown based on publicly available information on department websites). However, slightly less than one third of them (10/31) offer curriculum specifically about computational text analysis, suggesting that most of the formal training of sociologists in using NLP to analyze text is happening outside of sociology departments. Other social science departments also active in computational text analysis, such as psychology and political science, probably take on some of that burden by allowing students outside their respective departments enroll in their courses (one such example is noted in A1). But beyond these instances, it seems likely that the current cohorts of graduate students who are doing a lot of work energizing computational sociology are also forced to learn how to actually do the work from computer science and aligned programs, workshops, and/or internet based trainings outside of their home discipline. In principle, learning from computer scientists or experts from workshops is not a problem. However, machine learning sub-fields in computer science tend to create and then follow hype trends very closely, sometimes to their detriment lipton2019troubling . And the experts (along with computer scientists offering formal training) putting out training materials online or leading workshops probably have the skills and knowledge to identify issues when a particular method is not working well, something that students might find difficult to emulate themselves. Even beyond the issues related to technical training, crossing into disciplinary terrains could influence theoretical considerations, interpretation of results, and the very nature of the work. These trends have cascading effects as well, such as peer review and norm setting for the field.

Read the complete text