Automating Forecasting Question Generation and Resolution for AI Evaluation

Nikos I. Bosse, Peter Mühlbacher, Jack Wildman, Lawrence Phillips, Dan Schwarz · 2026

Read Automating Forecasting Question Generation and Resolution for AI Evaluation by Nikos I. Bosse, Peter Mühlbacher, Jack Wildman, Lawrence Phillips, Dan Schwarz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22444.txt.

Description

Nikos Bosse and four co-authors automate the making and grading of forecasting questions for AI tests. Forecasting — calling future events with odds — is prized for decisions and as a mark of general skill. Their 2026 preprint notes that judging AI forecasters needs many hard, varied questions, each later resolved with care.

Past automation leaned on a few repeating data feeds. The team aim for a broader, tougher stream of questions and firm resolutions. Good question farms let the field grade forecasters at scale. They also cut the human slog of test design. The work is infrastructure for AI evaluation.

Catalog record

CREATORNikos I. Bosse, Peter Mühlbacher, Jack Wildman, Lawrence Phillips, Dan Schwarz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE120 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22444

Browse the full library · About the Tower of Babel