RiddleBench: A New Generative Reasoning Benchmark for LLMs

Deepon Halder, Alan Saji Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan Raj Dabre · 2025

Read RiddleBench: A New Generative Reasoning Benchmark for LLMs by Deepon Halder, Alan Saji Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan Raj Dabre (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24932.txt.

Description

Deepon Halder, Alan Saji, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan, and Raj Dabre built RiddleBench, a set of hard riddles to test chat models. Known test sets lean on math and set skills, yet miss the loose, multi side thought that riddles need. Riddle work joins logic, space sense, and rule checks in one ask.

The bench holds more than seventeen hundred riddles in English. Top paid models scored just above sixty in a hundred on it. Checks showed deep flaws: models took bad thought from peers as true, and could not fix their own slips due to self trust bias. Their thought was also frail to small shifts in an ask.

Catalog record

CREATORDeepon Halder, Alan Saji Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan Raj Dabre
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24932

Browse the full library · About the Tower of Babel