DeonticBench : A Benchmark for Reasoning over Rules
Guangyao Dou ♡ Luis Brena ♡ Akhil Deo ♡ William Jurayj ♡ Jingyu Zhang ♡ · 2026
Read DeonticBench : A Benchmark for Reasoning over Rules by Guangyao Dou ♡ Luis Brena ♡ Akhil Deo ♡ William Jurayj ♡ Jingyu Zhang ♡ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04443.txt.
Description
Dou and colleagues introduce DEONTICBENCH, 6,232 tasks testing deontic reasoning about obligations, permissions and prohibitions under explicit rules. Long-context, high-stakes rule reasoning remains hard for language models, while most stress short math. Beyond chain-of-thought reasoning, the benchmark lets models translate statutes and case facts into executable Prolog, giving formal interpretations and traces. Across models, best hard-subset scores stay under 45 percent on SARA Numeric and 46.6 macro-F1 on Housing. Training with fine-tuning and reinforcement learning improves Prolog quality but still fails to solve the tasks.
Catalog record
| CREATOR | Guangyao Dou ♡ Luis Brena ♡ Akhil Deo ♡ William Jurayj ♡ Jingyu Zhang ♡ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72757 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04443