DeonticBench : A Benchmark for Reasoning over Rules

Guangyao Dou ♡ Luis Brena ♡ Akhil Deo ♡ William Jurayj ♡ Jingyu Zhang ♡ · 2026

Read DeonticBench : A Benchmark for Reasoning over Rules by Guangyao Dou ♡ Luis Brena ♡ Akhil Deo ♡ William Jurayj ♡ Jingyu Zhang ♡ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04443.txt.

Description

Dou and colleagues introduce DEONTICBENCH, 6,232 tasks testing deontic reasoning about obligations, permissions and prohibitions under explicit rules. Long-context, high-stakes rule reasoning remains hard for language models, while most stress short math. Beyond chain-of-thought reasoning, the benchmark lets models translate statutes and case facts into executable Prolog, giving formal interpretations and traces. Across models, best hard-subset scores stay under 45 percent on SARA Numeric and 46.6 macro-F1 on Housing. Training with fine-tuning and reinforcement learning improves Prolog quality but still fails to solve the tasks.

Catalog record

CREATORGuangyao Dou ♡ Luis Brena ♡ Akhil Deo ♡ William Jurayj ♡ Jingyu Zhang ♡
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72757
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04443

Browse the full library · About the Tower of Babel