Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments
Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger · 2026
Read Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments by Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30987.txt.
Introduction
Christopher Karvetski, Sheldon Huang and five colleagues test whether the quality of a forecaster's written reasoning can be scored by machine. Decision-makers rely on expert judgments paired with write-ups, but their quality resists measurement at scale. The team define sixty Explanation Quality Markers, reasoning patterns such as weighing base rates, scored by language models. The team ran a pre-registered analysis over 55,000 forecast-rationale pairs drawn from a tournament, and the markers predicted accuracy at both the single-forecast and forecaster levels, beating pre-LLM text-analysis methods.
Over 90% of the links ran the way the team predicted. The signal was one-sided: the markers flag likely weak forecasters more reliably than they separate the very best. Against standard skill measures they were the strongest predictor per forecast, though prior accuracy beat them per forecaster. Human quality ratings correlated less well and placed too much weight on length.
Catalog record
| CREATOR | Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas, Nadja Flechner, Jingyu Hu, Philip Tetlock, Ezra Karger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 179 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30987