LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

Juliusz Ziomek, William Bankes, Lorenz Wolf, Shyam Sundhar Ramesh, Xiaohang Tang, Ilija Bogunovic · 2026

Read LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs? by Juliusz Ziomek, William Bankes, Lorenz Wolf, Shyam Sundhar Ramesh, Xiaohang Tang, Ilija Bogunovic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-16902.txt.

Description

Juliusz Ziomek and co-authors introduce LLM-WikiRace, a benchmark for planning, reasoning, and world knowledge in large language models. Models must navigate Wikipedia hyperlinks step by step from a source page to a target page, requiring look-ahead planning and reasoning about how concepts connect in the real world. A broad set of open- and closed-source models is tested, including Gemini-3, GPT-5, and Claude Opus 4.5.

Those models achieve the strongest results on the easy level and show superhuman performance there. Yet scores drop sharply on hard difficulty: the best model, Gemini-3, succeeds in only about a quarter of hard games. World knowledge proves necessary for success, but only up to a point; beyond that threshold, planning and long-horizon reasoning dominate. Even the strongest models struggle to replan after failure, often entering loops rather than recovering. The benchmark is simple yet reveals clear limits in current reasoning systems.

Catalog record

CREATORJuliusz Ziomek, William Bankes, Lorenz Wolf, Shyam Sundhar Ramesh, Xiaohang Tang, Ilija Bogunovic
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-16902

Browse the full library · About the Tower of Babel