UQ : Assessing Language Models on Unsolved Questions
Fan Nie, Ken Ziyu Liu, Zihao Wang Rui Sun Wei Liu, Weijia Shi Huaxiu Yao Linjun Zhang Andrew Y. Ng, James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff · 2025
Read UQ : Assessing Language Models on Unsolved Questions by Fan Nie, Ken Ziyu Liu, Zihao Wang Rui Sun Wei Liu, Weijia Shi Huaxiu Yao Linjun Zhang Andrew Y. Ng, James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17580.txt.
Introduction
Fan Nie and co-authors present a benchmark built from questions no one has solved in this 2025 paper. Good tests for AI should be both hard and realistic. Exam-style tests can be made hard in artificial ways, while tests from real user chats tend to skew easy. Unsolved questions offer a way out of that tension.
Their testbed, called UQ, holds 500 hard and varied questions drawn from a public question site. Topics run from computer theory and math to science fiction and history. Questions are gathered with rule filters, model judges, and human review. Candidate answers are pre-screened by checker tools, then verified by the community over time. The platform stays open for experts to add and judge work.
Catalog record
| CREATOR | Fan Nie, Ken Ziyu Liu, Zihao Wang Rui Sun Wei Liu, Weijia Shi Huaxiu Yao Linjun Zhang Andrew Y. Ng, James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 188 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17580