Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation

Srujan P Mule, Aniketh Garikaparthi, Manasi Patwardhan · 2026

Read Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation by Srujan P Mule, Aniketh Garikaparthi, Manasi Patwardhan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21491.txt.

Introduction

Srujan P Mule and co-authors teach language models to forecast research success through comparative idea evaluation. Evaluating hundreds of AI-generated ideas without exhaustive experiments is a bottleneck. They ask whether models can forecast empirical success before experiments run. Given a goal and two candidate ideas, predict which performs better on benchmarks. On eleven thousand four hundred eighty-eight idea pairs, fine-tuning lifts accuracy to seventy-seven point one percent, beating GPT-5 at sixty-one point one percent.

Catalog record

CREATORSrujan P Mule, Aniketh Garikaparthi, Manasi Patwardhan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE138797
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21491

Browse the full library · About the Tower of Babel