ORFuzz : Fuzzing the "Other Side" of LLM Safety – Testing Over-Refusal
Haonan Zhang, Dongxia Wang *, Yi Liu, Kexin Chen, Jiashui Wang, Xinlei Ying, Long Liu, Wenhai Wang · 2025
Read ORFuzz : Fuzzing the "Other Side" of LLM Safety – Testing Over-Refusal by Haonan Zhang, Dongxia Wang *, Yi Liu, Kexin Chen, Jiashui Wang, Xinlei Ying, Long Liu, Wenhai Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11222.txt.
Description
Haonan Zhang and co-authors attack in 2025 the opposite failure of AI safety: models that refuse harmless questions. Over-refusal erodes trust, yet testing for it is weak, with flawed benchmarks and thin test generation. Their framework, ORFuzz, borrows fuzzing from software testing and aims it at language models.
Seed cases are picked by safety category, smart mutators rewrite them into fresh probes, and a judge model scores refusals the way users perceive them. In trials the fuzzer surfaced valid over-refusals at more than double the rate of leading baselines. Its cases form a public set of 1,855 tests that trip up ten models far more often than older sets.
Catalog record
| CREATOR | Haonan Zhang, Dongxia Wang *, Yi Liu, Kexin Chen, Jiashui Wang, Xinlei Ying, Long Liu, Wenhai Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-11222