UQ : Assessing Language Models on Unsolved Questions

Fan Nie, Ken Ziyu Liu, Zihao Wang Rui Sun Wei Liu, Weijia Shi Huaxiu Yao Linjun Zhang Andrew Y. Ng, James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff · 2025

Read UQ : Assessing Language Models on Unsolved Questions by Fan Nie, Ken Ziyu Liu, Zihao Wang Rui Sun Wei Liu, Weijia Shi Huaxiu Yao Linjun Zhang Andrew Y. Ng, James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17580.txt.

Introduction

Fan Nie and co-authors present a benchmark built from questions no one has solved in this 2025 paper. Good tests for AI should be both hard and realistic. Exam-style tests can be made hard in artificial ways, while tests from real user chats tend to skew easy. Unsolved questions offer a way out of that tension.

Their testbed, called UQ, holds 500 hard and varied questions drawn from a public question site. Topics run from computer theory and math to science fiction and history. Questions are gathered with rule filters, model judges, and human review. Candidate answers are pre-screened by checker tools, then verified by the community over time. The platform stays open for experts to add and judge work.

Catalog record

CREATORFan Nie, Ken Ziyu Liu, Zihao Wang Rui Sun Wei Liu, Weijia Shi Huaxiu Yao Linjun Zhang Andrew Y. Ng, James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE188 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17580

Browse the full library · About the Tower of Babel