TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models
Yiran Zhang, Mo Wang, Xiaoyang Li, Kaixuan Ren, Chencheng Zhu, Usman Naseem · 2025
Read TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models by Yiran Zhang, Mo Wang, Xiaoyang Li, Kaixuan Ren, Chencheng Zhu, Usman Naseem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01341.txt.
Description
Yiran Zhang and co-authors build TurnBench-MS, a test of multi-turn reasoning in language models in this 2025 paper. Most benchmarks ask one question at a time, unlike real work. Real problems demand guesses, feedback, and steady updating across rounds.
The game hides a rule that the model must crack through sequenced tries. Structured feedback after each guess forces belief revision over time. A classic mode covers standard play, while a nightmare mode stacks far harder chains. Step-level truth labels allow fine-grained scoring of the reasoning path. The best model hit high marks in classic play but collapsed in nightmare mode. Humans aced both, which shows the headroom left for models.
Catalog record
| CREATOR | Yiran Zhang, Mo Wang, Xiaoyang Li, Kaixuan Ren, Chencheng Zhu, Usman Naseem |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01341