From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs
Hanmeng Liu, Shichao Weng, Xiulai Li, Zhicai Zhang, Anli Yan, Xiaozhang Liu, [2pt] Hainan University, Haikou, China · 2026
Read From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs by Hanmeng Liu, Shichao Weng, Xiulai Li, Zhicai Zhang, Anli Yan, Xiaozhang Liu, [2pt] Hainan University, Haikou, China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07268.txt.
Description
Hanmeng Liu and five co-authors harden multiple-choice reasoning tests in a formal, validity-preserving way. Benchmarks saturate quickly and suffer contamination, while ad hoc tricks like paraphrasing add surface difficulty without logical rigor. LogiHard transforms simple selection questions into higher-order logical judgments, demanding more reasoning steps from the test taker.
An adaptive testing layer controls difficulty precisely with fewer questions. Across twelve leading models, hardened questions cut accuracy sharply, and transfer to a general benchmark showed a similar drop. The authors blame a completeness-checking gap in training rather than missing knowledge, a failure pattern human test takers did not share.
Catalog record
| CREATOR | Hanmeng Liu, Shichao Weng, Xiulai Li, Zhicai Zhang, Anli Yan, Xiaozhang Liu, [2pt] Hainan University, Haikou, China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07268