RuleArena : A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios
Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang · 2024
Read RuleArena : A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios by Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08972.txt.
Introduction
Following written rules sounds simple until the rules are baggage fees, basketball trade regulations, or tax law. Ruiwen Zhou and six co-authors build RuleArena, a benchmark that tests language models on exactly those three real-world rulebooks. The tasks demand long-context reading, logic, and exact arithmetic, going beyond the tidy logic puzzles of older tests.
Leading models did poorly overall. They picked the wrong rule among look-alikes, and even with the right rule in hand, their math often slipped. Giving the models external tools for arithmetic and logic lifted performance sharply. The authors read the gap as a research agenda: models must learn to find, apply, and compute with rules reliably. The benchmark and data are public, from a 2024 paper.
Catalog record
| CREATOR | Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08972
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RuleArena : A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang Abstract This paper introduces RuleArena , a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. Covering three practical domains—airline baggage fees, NBA transactions, and tax regulations— RuleArena assesses LLMs’ proficiency in handling intricate natural language instructions that demand long-context understanding, logical reasoning, and accurate mathematical computation. Two key attributes distinguish RuleArena from traditional rule-based reasoning benchmarks: (1) it extends beyond standard first-order logic representations, and (2) it is grounded in authentic, practical scenarios, providing insights into the suitability and reliability of LLMs for real-world applications. Our findings reveal several notable limitations in LLMs: (1) they struggle to identify and apply the appropriate rules, frequently becoming confused by similar but distinct regulations, (2) they cannot consistently perform accurate mathematical computations, even when they correctly identify the relevant rules, and (3) in general, they perform poorly in the benchmark. We also observe a significant performance boost when LLMs are provided with external tools for oracle math and logic operations. These results highlight significant challenges and promising research directions in advancing LLMs’ rule-guided reasoning capabilities in real-life applications. Our codes and data are publicly available on GitHub . 1 Introduction Figure 1: Overview of RuleArena . RuleArena contains 95 commonly used and moderately complex rules and 816 test problems from three representative real-world scenarios - airline luggage fees, NBA transactions, and taxation policies. LLMs are given a the task instruction, the reference rules in this scenario, and a user instance, and required to conduct reasoning and computation for the user input under the guidance of reference rules. Recently, Large Language Models (LLMs) ( Touvron et al., 2023 ; OpenAI, 2023 ; Team, 2023 ; Anthropic, 2024 ) have demonstrated remarkable capabilities across many real-world applications. However, their limited domain-specific knowledge often leads to unfaithful or misleading output, which can cause significant risks and financial liabilities. For example, Canadian airline was recently required to compensate a customer who received incorrect guidance from the airline’s chatbot 1 1 1 https://www.theguardian.com/world/2024/feb/16/air-canada-chatbot-lawsuit . These challenges highlight the need for robust, real-world benchmarks that assess how faithfully and accurately LLMs can follow real-life instructions and adhere to relevant regulations, thereby ensuring reliable and safe outputs for deployment. Although several studies have examined LLMs’ instruction-following abilities ( Chen et al., 2024a ; Jiang et al., 2024 ; Wen et al., 2024 ) , they mainly focused on style constraints, such as the format Zhou et al. (2023) , length, or topic of responses. Yet, the significance of instruction-following extends well beyond style compliance. In many problem-solving scenarios, instructions function as rules : they impose logical constraints on the reasoning process and specify how answers should be derived from given inputs. However, limited attention has been paid to LLMs’ capacity to follow complex rules. Existing research ( Mu et al., 2023 ; Sun et al., 2024 ) largely addresses only single-step, first-order logic reasoning or artificially synthesized logical tasks. In contrast, real-world rules frequently appear in diverse and nuanced natural language forms. They may involve intricate logical structures, including the need for parallel reasoning across multiple rules or navigating interdependent rule sets. For instance, to calculate the fees for checked luggage when taking flights, one needs to consider the base price for checking each item, the overweight and oversize charges, and how these charges should be aggregated together. The extent to which LLMs can accurately follow these complex, real-world rules—an ability we term rule-guided reasoning —remains unknown. To better understand the complexity and practical implications of rule-guiding reasoning, we introduce a new evaluation benchmark, RuleArena , grounded in realistic scenarios. As illustrated in Figure 1 , RuleArena is developed from three representative, real-world scenarios: (1) airline luggage fee policies, (2) NBA transactions, and (3) taxation policies. From these domains, we collect authentic rules currently implemented by companies or government agencies. For each domain, we construct a set of challenging test problems, pairing each question with a ground-truth solution, and then evaluate a range of state-of-the-art LLMs on their ability to conform to the rules. LLMs are provided with the domain-specific task instructions and reference rules, and required to resolve each test problem through reasoning according to the question and reference rules. Our contributions in this work can be summarized in three main points: • A diverse collection of real-world rules: We assemble a comprehensive set of 95 policies/rules drawn from these three real-world scenarios. • A challenging benchmark and novel evaluation metrics: Using the collected rules, we introduce RuleArena , a new benchmark containing 816 datapoints designed to test LLMs’ rule-guided reasoning ability. We further propose a suite of evaluation metrics for both rule selection and rule application, providing fine-grained insights into LLMs’ performance. • A comprehensive analysis of prevalent challenges: By examining common failure cases, identifying difficult rule types, and conducting extensive controlled experiments, we uncover several systematic issues that limit current LLMs’ rule-guided reasoning capabilities and promising directions to improve. 2 Related Work Complex Instruction-following Benchmarks A wide range of benchmarks has been designed to evaluate LLMs’ instruction-following abilities from various perspectives, including semantics ( Zheng et al., 2023 ; Li et al., 2023 ; Liu et al., 2023 ; Wu et al., 2024a ; Wu et al., 2024b ) , format ( Xia et al., 2024 ; Tang et al., 2023 ) , and response length ( Chen et al., 2024b ; Sun et al., 2023 ) . To further probe complexity, some works have introduced benchmarks that construct complex instructions through compositional methods. For example, WizardLM ( Xu et al., 2023 ) generates intricate tasks by combining simpler instructions, while CELLO ( He et al., 2024 ) uses task descriptions and input texts to create complex prompts grounded in real-world scenarios. ComplexBench ( Wen et al., 2024 ) adopts multiple compositional structures to integrate atomic requirements into more challenging instructions. In contrast, our work focuses on instructions derived directly from real-life scenarios, where naturally occurring complexities arise from multifaceted constraints incurred by inputs on the set of instructions. Logical Reasoning Benchmarks Extensive research has explored benchmarks for mathematical ( Koncel-Kedziorski et al., 2016 ; Ling et al., 2017 ; Amini et al., 2019 ; Cobbe et al., 2021 ; Hendrycks et al., 2021 ) and logical ( Mao et al., 2019 ; Gupta et al., 2020 ; Tafjord et al., 2021 ; Zhong et al., 2022 ; Han et al., 2022 ; Zhang and Ding, 2024 ) reasoning, evaluating LLMs’ abilities to solve math problems of varying difficulty, tackle coding challenges, and engage in deductive logic. Although these benchmarks test models’ reasoning skills, their logical constraints are often represented in simplified, formal systems, such as propositional ( Hua et al., 2024 ) or first-order logic ( Zhu et al., 2023 ; Mu et al., 2023 ; Sun et al., 2024 ) . In contrast, our benchmark deals with rules that arise in natural language, capturing a richer, more realistic set of constraints. Such natural language rules extend beyond neatly formalized logical representations, often express higher-order logic