pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs

Tobias Schimanski, Imene Kolli, Yu Fan, Ario Saeid Vaghefi, Jingwei Ni, Elliott Ash, Markus Leippold · 2026

Read pdfQA: Diverse, Challenging, and Realistic Question Answering over PDFs by Tobias Schimanski, Imene Kolli, Yu Fan, Ario Saeid Vaghefi, Jingwei Ni, Elliott Ash, Markus Leippold (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-02285.txt.

Description

Tobias Schimanski and six co authors introduce pdfQA, a data set for testing whether computer systems can answer questions about real PDF files. PDFs are the second most used document type on the web, yet most question answering sets start from plain text or cover one narrow field. A PDF adds layout, tables, and mixed sources that plain text sets miss.

The set holds about two thousand human written question and answer pairs and a similar synthetic set, spread across many domains. Each pair is tagged along ten dimensions of difficulty, such as file type, where the answer sits, and what kind of answer is needed. Tests with open language models show clear failures that line up with those difficulty tags, making the set useful for checking full reading pipelines.

Catalog record

CREATORTobias Schimanski, Imene Kolli, Yu Fan, Ario Saeid Vaghefi, Jingwei Ni, Elliott Ash, Markus Leippold
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-02285

Browse the full library · About the Tower of Babel