To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation
Xiang Cheng, Raveesh Mayya, João Sedoc, Updated: 2 April 2026. (For the Latest Version: Click Here ) · 2024
Read To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation by Xiang Cheng, Raveesh Mayya, João Sedoc, Updated: 2 April 2026. (For the Latest Version: Click Here ) (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14461.txt.
Introduction
Xiang Cheng, Raveesh Mayya, and Joao Sedoc address reproducibility in language model annotation in a 2024 paper. Management research depends on labeling unstructured text, and large models offer a cheap, scalable annotator. But proprietary models get retired, so results built on them may become impossible to reproduce.
The authors split measurement error into four sources: guidelines, human baselines, prompts, and model differences. Their SILICON workflow prescribes fixes at each source. Tests across nine management tasks showed the fixes reduce error, and simulations showed cleaner downstream statistics. Further steps target reproducibility once error is controlled.
Catalog record
| CREATOR | Xiang Cheng, Raveesh Mayya, João Sedoc, Updated: 2 April 2026. (For the Latest Version: Click Here ) |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 139 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14461