ORFuzz : Fuzzing the "Other Side" of LLM Safety – Testing Over-Refusal

Haonan Zhang, Dongxia Wang *, Yi Liu, Kexin Chen, Jiashui Wang, Xinlei Ying, Long Liu, Wenhai Wang · 2025

Read ORFuzz : Fuzzing the "Other Side" of LLM Safety – Testing Over-Refusal by Haonan Zhang, Dongxia Wang *, Yi Liu, Kexin Chen, Jiashui Wang, Xinlei Ying, Long Liu, Wenhai Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11222.txt.

Description

Haonan Zhang and co-authors attack in 2025 the opposite failure of AI safety: models that refuse harmless questions. Over-refusal erodes trust, yet testing for it is weak, with flawed benchmarks and thin test generation. Their framework, ORFuzz, borrows fuzzing from software testing and aims it at language models.

Seed cases are picked by safety category, smart mutators rewrite them into fresh probes, and a judge model scores refusals the way users perceive them. In trials the fuzzer surfaced valid over-refusals at more than double the rate of leading baselines. Its cases form a public set of 1,855 tests that trip up ten models far more often than older sets.

Catalog record

CREATORHaonan Zhang, Dongxia Wang *, Yi Liu, Kexin Chen, Jiashui Wang, Xinlei Ying, Long Liu, Wenhai Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11222

Browse the full library · About the Tower of Babel