A Benchmark for Deep Information Synthesis

Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Jun Wang, Gerasimos Lampouras Dataset Contributors, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan, Alex Sivakumar, Philipp Borchert, Ismail Elezi, Meiru Zhang, Ka Yiu Lee, Guchun Zhang · 2026

Read A Benchmark for Deep Information Synthesis by Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Jun Wang, Gerasimos Lampouras Dataset Contributors, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan, Alex Sivakumar, Philipp Borchert, Ismail Elezi, Meiru Zhang, Ka Yiu Lee, Guchun Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21143.txt.

Description

Language-model agents increasingly tackle research-like tasks, but benchmarks rarely test synthesis across sources. Debjit Paul and sixteen co-authors introduce such a benchmark in this 2026 arXiv paper: DEEPSYNTH, with one hundred twenty tasks across seven domains covering sixty-seven countries. Annotators gathered official sources, formed hypotheses, analyzed manually, and designed tasks with verifiable answers.

Eleven leading models and deep-research agents score at most just under nine on F1 and seventeen and a half on a judge metric, showing how hard synthesis remains. Agents struggle with hallucinations and reasoning over large information spaces. The benchmark gives the field a realistic yardstick for guiding research beyond simple fact retrieval.

Catalog record

CREATORDebjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov, Jun Wang, Gerasimos Lampouras Dataset Contributors, Lena Sophia Bolliger, Aysim Toker, Roy Miles, Andreea-Maria Oncescu, Jasivan, Alex Sivakumar, Philipp Borchert, Ismail Elezi, Meiru Zhang, Ka Yiu Lee, Guchun Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-21143

Browse the full library · About the Tower of Babel