RuozhiBench : Evaluating LLMs with Logical Fallacies and Misleading Premises
Zenan Zhai, Hao Li, Xudong Han, Zhenxuan Zhang, Yixuan Zhang, Timothy Baldwin, Haonan Li · 2025
Read RuozhiBench : Evaluating LLMs with Logical Fallacies and Misleading Premises by Zenan Zhai, Hao Li, Xudong Han, Zhenxuan Zhang, Yixuan Zhang, Timothy Baldwin, Haonan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13125.txt.
Introduction
Zhai and colleagues wrote in twenty twenty-five a test of whether language models spot tricks in questions. Models can answer hard reasoning questions, yet fallacies and false premises are less studied. The team built RuozhiBench, a set of six hundred seventy-seven questions in two languages. Each item hides a logical fallacy or a misleading starting claim. Human experts crafted and reviewed the items with care.
They tested seventeen models from five families in two answer formats. Even strong models did poorly at spotting the tricks. The best model reached only about sixty-two percent correct. Human scores passed ninety percent on the same material. The gap shows a real weakness in current models. The data set gives builders a clear target for improvement.
Catalog record
| CREATOR | Zenan Zhai, Hao Li, Xudong Han, Zhenxuan Zhang, Yixuan Zhang, Timothy Baldwin, Haonan Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13125
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RuozhiBench : Evaluating LLMs with Logical Fallacies and Misleading Premises Zenan Zhai, Hao Li, Xudong Han, Zhenxuan Zhang, Yixuan Zhang, Timothy Baldwin, Haonan Li Abstract Recent advances in large language models (LLMs) have shown that they can answer questions requiring complex reasoning. However, their ability to identify and respond to text containing logical fallacies or deliberately misleading premises remains less studied. To address this gap, we introduce RuozhiBench , a bilingual dataset comprising 677 carefully curated questions that contain various forms of deceptive reasoning, meticulously crafted through extensive human effort and expert review. In a comprehensive evaluation of 17 LLMs from 5 Series over RuozhiBench using both open-ended and two-choice formats, we conduct extensive analyses on evaluation protocols and result patterns. Despite their high scores on conventional benchmarks, these models showed limited ability to detect and reason correctly about logical fallacies, with even the best-performing model, Claude-3-haiku , achieving only 62% accuracy compared to the human of more than 90%. 1 1 1 Data and code available at: https://github.com/LibrAIResearch/ruozhibench 2 2 2 Data license: CC-BY-NC license. 1 Introduction Large language models (LLMs) have rapidly advanced in recent years, demonstrating impressive capabilities across a wide range of tasks ( Zhang et al., 2022 ; Scao et al., 2022 ; Touvron et al., 2023 ; Bai et al., 2023a ; DeepSeek-AI et al., 2025 ) . Benchmarking plays a crucial role in assessing their performance, with existing evaluations falling into two main categories. The first category includes benchmarks with predefined answers, such as MMLU ( Hendrycks et al., 2021a ; Li et al., 2023a ) and ARC ( Clark et al., 2018 ) , which assess factual knowledge and reasoning. However, issues like data contamination ( Carlini et al., 2022 ; Sainz et al., 2023 ) and potential model cheating have been widely reported. The second category evaluates open-ended responses, focusing on alignment with human preferences. AlpacaEval ( Li et al., 2023b ) is a widely-used automatic evaluator that measures model performance by comparing outputs against a reference model, typically using GPT-4 as the judge. While effective for general instruction following, it is not designed to assess how models handle deceptive or misleading inputs. However, in real-world scenarios, texts often contain logical fallacies, misleading premises, or intentional ambiguities that can trap the unwary. When faced with such deceptive inputs, current models often fail to identify the underlying fallacies and provide responses that reveal their limitations in logical reasoning (Figure 1 ). Despite these types of deliberately misleading texts being a crucial test of true language understanding, there is currently no benchmark specifically designed to evaluate how well models handle such scenarios. Figure 1: An example of a question from RuozhiBench and response from GPT-4o . Figure 2: The creation process for RuozhiBench , consisting of three main parts: data filtering (left), translation and review (middle), and annotation (right). To address this gap, we introduce RuozhiBench , a novel benchmark designed to evaluate the ability of models to identify and reason about deceptive inputs and logical fallacies. RuozhiBench comprises 677 questions sourced from the Chinese forum Ruozhiba , a platform which contains texts that appear reasonable at first glance but contain subtle logical traps or misleading premises. To ensure high data quality, we implemented rigorous filtering, preprocessing, and annotation. Each question was carefully reviewed and translated into English while preserving its deceptive nature. We then systematically categorized the questions into six distinct types, ensuring clear and consistent labeling. See Section 2 for more details. To further enhance reliability, we designed a multi-step annotation process involving both human validation and automated checks. Only questions that met strict criteria for clarity, difficulty, and linguistic adaptation were included. Additionally, we conducted both rating-based and selection-based evaluations, using human judgments as a reference, and employed multiple automated evaluation methods to measure model performance. Our preliminary experiments assessed 17 LLMs, revealing a substantial gap between model performance and the human upper bound. Despite achieving high scores on standard benchmarks, these models still lag behind humans in logical reasoning and fallacy detection. RuozhiBench is a critical step towards a more comprehensive assessment of models’ ability to handle deceptive inputs and logical fallacies. 2 RuozhiBench-Gen 2.1 Data Source Ruozhiba (literally meaning ‘‘moron forum’’) is one of the most popular online forums in the Chinese internet community, known for its collection of brain teasers, logical puzzles, and deliberately misleading questions. The forum’s content often features unconventional perspectives and clever wordplay that challenges conventional thinking patterns. Our work begins with the raw data collected by a previous project, 3 3 3 https://github.com/Leymore/ruozhiba which compiled a comprehensive collection of threads from Ruozhiba . 4 4 4 Note that Baidu Tieba content is freely available for academic research purposes with no legal restrictions. ID Category # Q. Description Example 1 Logical Error 142 When the question contains logical contradictions or reasoning errors, including violations of logical rules, making it logically untenable. I pressed the mute button on my laptop, why is the fan still so loud? 2 Commonsense Misunderstanding 526 The question reflects a misunderstanding of basic common sense or universally accepted facts, usually involving incorrect interpretations of daily knowledge. Is it better to prevent tooth decay by applying toothpaste directly to the teeth without brushing before going to bed? 3 Erroneous Assumption 471 The question is based on one or more incorrect assumptions, leading to inaccuracies in the question or its answer. If you stretch your leg to trip a moving car, will it overturn? 4 Scientific Misconception 30 The question involves misunderstandings of scientific principles or knowledge, including incorrect interpretations of scientific theories or methods. Can you avoid drone thermal imaging bombings by eating only low-calorie foods? 5 Absurd Imagination 463 The question setting is contrary to reality or common sense, containing impossible or illogical elements. If you suck away all the clouds, will it stop raining and be sunny forever? 6 Others 17 If the provided categories do not match the current question, please choose this option. Oxygen can rust iron. Our blood contains iron, why doesn’t our blood rust? Table 1: Classification schema of deceptive questions: categories, descriptions, and examples. Note that a given question may belong to multiple categories. 2.2 Data Screening From the initial 86,000 entries, we first extracted over 8,000 interrogative sentences using string matching. We then implemented a rigorous filtering process with three annotators with humanities backgrounds. They first removed questions with heavy cultural dependencies or potentially negative influences, reducing the dataset to 820 entries. Through collaborative review and discussion, the annotators further filtered questions based on their suitability for English translation and comprehension, removing entries where translation would significantly alter the original meaning or logical structure. This process yielded our final dataset of 677 questions, ensuring each entry maintains its original logical challenge while being accessible to a global audience. 2.3 Data Annotation After data screening, we conducted four rounds of annotation for these questions: translation review, paired question generation, irrationality analysis, and question type categorization. For all steps except paired question generation, we employed a hybrid approach combining LLM-based initial processing with human verification. The annotators involved had both bilingual (Chinese–English) and NLP backgrounds. Translation Review In the translation stage, we first used Google