Researchers waste 80% of LLM annotation costs by classifying one text at a time †
Christian Pipal, Eva-Maria Vogel, Morgan Wack, Frank Esser, Department of Communication, Media Research (IKMZ), University of Zurich · 2026
Read Researchers waste 80% of LLM annotation costs by classifying one text at a time † by Christian Pipal, Eva-Maria Vogel, Morgan Wack, Frank Esser, Department of Communication, Media Research (IKMZ), University of Zurich (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03684.txt.
Introduction
Christian Pipal and colleagues showed that coding texts one at a time wastes most of the money spent on language-model annotation. Grouping 25 items per prompt and stacking all variables together cut the calls needed for 100,000 texts on four variables from 400,000 to 4,000. Tests on eight models found accuracy held up through batches of 100, so the savings cost little quality.
Catalog record
| CREATOR | Christian Pipal, Eva-Maria Vogel, Morgan Wack, Frank Esser, Department of Communication, Media Research (IKMZ), University of Zurich |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17905 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Researchers waste 80% of LLM annotation costs by classifying one text at a time † Christian Pipal, Eva-Maria Vogel, Morgan Wack, Frank Esser, Department of Communication, Media Research (IKMZ), University of Zurich Abstract Large language models (LLMs) are increasingly being used for text classification across the social sciences, yet researchers overwhelmingly classify one text per variable per prompt. Coding 100,000 texts on four variables requires 400,000 API calls. Batching 25 items and stacking all variables into a single prompt reduces this to 4,000 calls, cutting token costs by over 80%. Whether this degrades coding quality is unknown. We tested eight production LLMs from four providers on 3,962 expert-coded tweets across four tasks, varying batch size from 1 to 1,000 items and stacking up to 25 coding dimensions per prompt. Six of eight models maintained accuracy within 2 pp of the single-item baseline through batch sizes of 100. Variable stacking with up to 10 dimensions produced results comparable to single-variable coding, with degradation driven by task complexity rather than prompt length. Within this safe operating range, the measurement error from batching and stacking is smaller than typical inter-coder disagreement in the ground-truth data. † † footnotetext: C.P. designed research, performed research, analyzed data, and wrote the paper. E.-M.V., M.W., and F.E. contributed to research design and edited the paper. The authors declare no competing interest. Keywords: large language models, automated content analysis, batch prompting, text classification, computational methods Introduction Large language models (LLMs) have quickly become standard tools for automated content analysis across the social sciences ( 2 ) , and are now used to classify millions of social media posts ( 8 ) , code news content at scale ( 14 ) , and score hundreds of thousands of academic texts ( 11 ) . Validation work has established that LLMs match or exceed human coders on tasks central to this research ( 7 ; 16 ; 12 ) . A methodological literature has emerged to guide prompt design and model selection ( 3 ; 4 ) . Yet two implementation choices that shape every LLM-based project remain untested. Item batching determines how many texts are grouped in a single prompt, while variable stacking determines how many coding dimensions are requested per call. Published practice is nearly universal in its conservatism, with one text, one variable, and one prompt per API call ( 7 ; 16 ; 9 ) . Coding 100,000 texts on four variables at this baseline requires 400,000 API calls. Batching 25 items and stacking all four variables reduces this to 4,000 calls with over 80% cost savings, because instructions are transmitted once rather than repeated per call. For projects of the scale described above, these choices can determine the feasibility of a study. The computer science literature shows that batching is viable at small sizes on standard NLP benchmarks ( 5 ; 10 ) , and that multi-task inference is architecture-dependent ( 13 ; 6 ) . These findings do not address codebook-driven classification typical in social science content analysis. Whether batching and stacking degrade quality in this setting, and whether their effects compound, is an open question. We provide the first systematic test, varying batch size from 1 to 1,000 and stacking up to 25 variables across eight production LLMs and 960,000 classifications. Results Item batching. We LLM-coded 3,962 expert-labeled tweets ( 7 ) on four variables (relevance, problem/solution, stance, topic) across nine batch sizes ( b\in\{1,5,10,25,50,100,250,500,1000\} ) using eight LLMs from four providers. Accuracy remained stable for most models through b=100 (Fig. 1 A). Six of eight models showed accuracy drops below 2 pp between b=1 and b=100 while saving over 80% in token costs. Claude Haiku 4.5 was the most robust, with a maximum drop of 1.1 pp across all batch sizes. Degradation at b\geq 250 was model-specific. Google and Alibaba models declined 3–4 points by b=1000 . Two OpenAI reasoning models (GPT-5-nano, GPT-5-mini), which cannot run at temperature 0, collapsed at these sizes, losing 27–36 points with substantial parse failures. No other model exceeded 1.5% parse failure at any batch size. The four variables responded differently to batching (Fig. 2 ). Relevance (binary, N=3{,}179 ) was the most sensitive in relative terms, with most models losing 5–9 pp by b=1000 , though absolute accuracy remained above 83% for all non-OpenAI models. Problem/solution (three-class) showed moderate degradation of 2–5 points. Stance (three-class, N=783 ) was noisy throughout due to class imbalance. Topic (six-class, N=611 ) showed a different pattern. Accuracy increased at larger batch sizes for several models, possibly because co-presented tweets provide distributional information that helps calibrate category boundaries. A: Item Batching B: Variable Stacking Figure 1: Batching and stacking are safe within bounds. ( A ) Cost savings (%) vs. accuracy change (pp) relative to b=1 for each model at each batch size. The dashed line traces the mean trajectory across all eight models; the shaded region marks the min–max range. Batch size labels mark progression along this trajectory. Six models remain near zero accuracy loss through b=100 ( {\sim}84\% cost savings). Two OpenAI reasoning models (green) collapse at b\geq 250 , driving the shaded region downward. ( B ) Overall accuracy (mean across four variables) under stacking conditions at b=25 . k=1 : single-variable baseline from Study 1. k=4,10,25 : number of simultaneous coding dimensions in Study 2. Control (orange): same prompt length as k=25 but only four variables plus filler text, showing that degradation reflects task complexity, not prompt length. Variable stacking. We crossed three levels of stacking ( k\in\{4,10,25\} simultaneous coding dimensions) with batch size b=25 , comparing against the single-variable baseline ( k=1 ) from Study 1 (Fig. 1 B). Moving from k=1 to k=25 reduced accuracy by 1.6–6.1 pp for seven of eight models, though GPT-5-nano dropped 15.5 points. Stacking up to k=10 produced results within 2.2 pp of the single-variable baseline for all models except GPT-5-nano. The batching-stacking interaction was additive rather than multiplicative. Accuracy at b=250 showed comparable stacking effects, so researchers can optimize along both dimensions without a compounding penalty. Figure 2: Accuracy by model, variable, and batch size (Study 1). Each panel shows one of the four variables. Bars represent batch sizes from b=1 (darkest) to b=1000 (lightest). Six of eight models maintain stable accuracy through b=100 across all four variables. GPT-5 Nano shows progressive degradation starting at b=5 . Topic accuracy increases with batch size for several models, possibly reflecting an auto-demonstration effect in which co-presented tweets provide implicit distributional information. Prompt length vs. task complexity. A control prompt matching the token length of the k=25 condition but containing only four ground-truth variables (with background text filling the remaining space) performed at the level of k=4 for all models (Fig. 1 B, orange bars). Degradation under stacking is caused by the demand of managing multiple classification schemes simultaneously, not by additional tokens in the prompt. Discussion Batch sizes of 25–100 are safe for most current LLMs, saving over 80% in costs with accuracy loss below 2 pp. Stacking up to 10 coding dimensions is also viable, with degradation driven by task complexity rather than prompt length. Model choice matters more than batch size. At any given b , the gap between