To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation

Xiang Cheng, Raveesh Mayya, João Sedoc, Updated: 2 April 2026. (For the Latest Version: Click Here ) · 2024

Read To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation by Xiang Cheng, Raveesh Mayya, João Sedoc, Updated: 2 April 2026. (For the Latest Version: Click Here ) (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14461.txt.

Introduction

Xiang Cheng, Raveesh Mayya, and Joao Sedoc address reproducibility in language model annotation in a 2024 paper. Management research depends on labeling unstructured text, and large models offer a cheap, scalable annotator. But proprietary models get retired, so results built on them may become impossible to reproduce.

The authors split measurement error into four sources: guidelines, human baselines, prompts, and model differences. Their SILICON workflow prescribes fixes at each source. Tests across nine management tasks showed the fixes reduce error, and simulations showed cleaner downstream statistics. Further steps target reproducibility once error is controlled.

Catalog record

CREATORXiang Cheng, Raveesh Mayya, João Sedoc, Updated: 2 April 2026. (For the Latest Version: Click Here )
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE139 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14461

Browse the full library · About the Tower of Babel