Consistency Checks for Language Model Forecasters

Daniel Paleka, Abhimanyu Pallavi Sudhir *, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, Florian Tramèr · 2024

Read Consistency Checks for Language Model Forecasters by Daniel Paleka, Abhimanyu Pallavi Sudhir *, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, Florian Tramèr (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18544.txt.

Introduction

Seven authors led by Daniel Paleka grade forecast models by whether their calls hang together, since truth lands too late to grade by. A call set that fights itself can be milked by a sharp trader for sure gain. Their gauge counts that free gain: call both sides of one race at sixty percent, and the house bleeds. An auto rig builds base asks, spins logic kin asks, and takes the model calls on each.

Those snap grades track the slow true scores of the models in a side bench. A long board that pays out in 2028 ships with the work for the field. Fast, fair grades can now steer forecaster builds in real time. Logic grit shows as a skill in its own right.

Catalog record

CREATORDaniel Paleka, Abhimanyu Pallavi Sudhir *, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, Florian Tramèr
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE156 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18544

Browse the full library · About the Tower of Babel