FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
Zhehao Zhang, Weijie Xu, Fanyou Wu, Chandan K. Reddy · 2025
Read FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning by Zhehao Zhang, Weijie Xu, Fanyou Wu, Chandan K. Reddy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08054.txt.
Description
Zhehao Zhang and three co-authors released a resource in a 2025 paper to curb needless refusals by language models. Safety tuning often overshoots: models refuse harmless questions that merely sound risky, which guts their usefulness in delicate settings. FalseReject holds 16,000 seemingly toxic queries across 44 safety categories, each paired with a structured, reasoning-first response that shows why the request is safe or how to answer carefully.
A graph-guided team of adversarial agents generated the varied prompts, and human annotators built the test benchmark. Tests across 29 leading models showed over-refusal is widespread. Fine-tuning on the new data cut needless refusals sharply, without weakening overall safety or general language skill. Training sets serve both standard and reasoning-focused models.
Catalog record
| CREATOR | Zhehao Zhang, Weijie Xu, Fanyou Wu, Chandan K. Reddy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08054