Much Ado about Prompting: LLM classification of text messages from experiments
Can Çelebi, Stefan P Penczynski · 2026
Read Much Ado about Prompting: LLM classification of text messages from experiments by Can Çelebi, Stefan P Penczynski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/plos-pone-0354757.txt.
Description
Can Çelebi and Stefan Penczynski test in 2026 how well human codebooks work as prompts when large language models classify text from experiments. Three codebooks from experimental economics were used, covering promise keeping and strategic thinking. Prompt detail, format, framing, and wording were varied across two commercial and two open models, and results were compared with human annotators.
Codebooks as prompts agreed with humans at a high rate across tasks. On a recognition heavy task, model choice drove accuracy most, while on learning heavy tasks, instruction detail mattered as much. Extra content helped larger models but could hurt smaller ones. Write prompts like good human instructions, the authors urge.
Catalog record
| CREATOR | Can Çelebi, Stefan P Penczynski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53648 |
| LICENSE | CC_BY |
| RIGHTS | CC BY |
| SOURCE | PLOS |