DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities

Tianyi Zhuang, Chuqiao Kuang, Xiaoguang Li, Yihua Teng, Jihao Wu, Yasheng Wang, Lifeng Shang · 2025

Read DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities by Tianyi Zhuang, Chuqiao Kuang, Xiaoguang Li, Yihua Teng, Jihao Wu, Yasheng Wang, Lifeng Shang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17807.txt.

Description

Tianyi Zhuang and co-authors bench mark long document reasoning in a 2025 paper. Real tasks ask models to chain steps across long, messy files. Most tests score just the final reply, so lucky guesses pass as thought.

DocPuzzle holds one hundred expert tasks built with a human and AI tag team. A check list grades the steps of each reply, not just its end. Slow think models beat plain chat models by a clear mark. Small distilled models fell far behind their teacher, a warn that copying replies drops true skill. Step grades give a truer view of machine thought.

Catalog record

CREATORTianyi Zhuang, Chuqiao Kuang, Xiaoguang Li, Yihua Teng, Jihao Wu, Yasheng Wang, Lifeng Shang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17807

Browse the full library · About the Tower of Babel