FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning Zhehao Zhang, Weijie Xu, Fanyou Wu, Chandan K. Reddy Abstract Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities. Content Warning : This paper contains discussions of controversial or potentially unsafe content as examples. 1 Introduction As large language models (LLMs) become widely integrated into real-world applications, balancing their safety and helpfulness remains challenging ( Askell et al. 2021 ) . Considerable efforts have aimed to enhance model helpfulness for example by improving reasoning for complex queries ( Wei et al. 2022 ) , while extensive research has also focused on aligning models to refuse unsafe requests ( Bai et al. 2022 ) . However, these safety measures unintentionally introduce over-refusal, 1 1 1 Also known as exaggerated safety or false rejection; we use ”over-refusal” in this paper. where models unnecessarily reject benign instructions, negatively impacting user experiences, particularly in sensitive scenarios (see Figure 1 ). Previous research ( Röttger et al. 2024 ) has investigated the over-refusal problem by developing datasets of adversarial prompts that appear unsafe but are actually benign, aiming to benchmark LLM performance. While most of these datasets are manually written by humans, the rapid progress of LLMs has made them less challenging for SOTA models ( Cui et al. 2024 ) . Existing synthetic data generation approaches are also inadequate, as they rely on simple sampling strategies and lack robust mechanisms to generate diverse and complex over-refusal queries at scale. Additionally, emerging large reasoning models, such as OpenAI-o1 ( Jaech et al. 2024 ) and DeepSeek-R1 ( Guo et al. 2025 ) , have exhibited safety issues ( Zhou et al. 2025 ) but remain underexplored regarding their susceptibility to over-refusal. Most importantly, previous datasets focus solely on evaluation and do not provide training sets that could help calibrate models’ over-refusal behavior. A fundamental challenge in mitigating LLMs’ over-refusal behavior lies in the inherent ambiguity of natural language. Prior research ( An et al. 2024b ) has identified many scenarios where prompts are neither clearly harmful nor clearly safe. These controversial prompts complicate evaluation, raising open questions about whether binary compliance or refusal decisions are adequate. They also pose challenges for calibration, particularly in determining how models can generate appropriate and informative responses instead of direct refusal, while still ensuring safety. To address these challenges, we introduce FalseReject , a large-scale resource designed to systematically evaluate and calibrate over-refusal behaviors in LLMs across various safety scenarios. Unlike previous datasets that primarily rely on manually crafted prompts or simple sampling data generation, we generate complex and diverse over-refusal queries through a graph-informed adversarial multi-agent interaction framework . Specifically, we iteratively refine generated queries by leveraging dynamic role-play interactions between two LLM-based agents, guided by validation feedback from a pool of LLM evaluators. This structured iterative refinement, informed by entity graphs extracted from safety-related datasets, ensures the generated queries exhibit sufficient complexity and realism to effectively challenge current SOTA LLMs. To construct responses for our queries, we design structured responses that explicitly incorporate safety reasoning, enabling models to distinguish between safe and unsafe contexts in controversial scenarios. Besides, to benchmark the over-refusal behaviors of SOTA LLMs, we created a high-quality test set carefully filtered through human annotation. The main contributions are summarized as follows: • A novel graph-informed adversarial multi-agent interaction framework that effectively generates diverse and challenging over-refusal queries at scale. • A novel dataset (FalseReject) containing 15k training examples (FalseReject-Train-Instruct and FalseReject-Train-CoT) and 1.1k test examples (FalseReject-Test), spanning 44 safety-related categories, for both standard and reasoning-oriented LLMs. • Comprehensive benchmarking across 29 SOTA LLMs, revealing significant variability in over-refusal behaviors among different model families. • Empirical validation showing that supervised finetuning (SFT) on our dataset substantially improves models’ abilities to distinguish safe from unsafe contexts, reducing unnecessary refusals without harming general language capabilities, and achieving deeper, more consistent alignment compared to standard instruction-tuned models. Figure 1: Examples include a non-reasoning LLM that directly refuses a benign prompt and a reasoning model that fully complies without considering safety. In contrast, models fine-tuned with our FalseReject dataset can effectively distinguish between safe and unsafe contexts and provide helpful information while maintaining safety. 2 Related Work Dataset Size Topics Train LLM-Gen Rejection Rate Self-BLUE \downarrow Dist-2 \uparrow CoT XSTest ( Röttger et al. 2024 ) 250 18 ✗ ✗ 12.10 0.21 0.69 ✗ OKTest ( Shi et al. 2024 ) 350 18 ✗ ✗ 19.75 0.31 0.64 ✗ PHTest ( An et al. 2024a ) 3,260 10 ✗ ✓ 14.00 0.40 0.52 ✗ OR-Bench ( Cui et al. 2024 ) 80K 10 ✗ ✓ 6.20 0.35 0.53 ✗ FalseReject (Ours) 16K 44 ✓ ✓ 40.46 0.26 0.65 ✓ Table 1: Comparison of FalseReject with existing over-refusal datasets. We bold the best scores for both LLM-generated and human-written ones. Topics indicate the number of sensitive topic categories covered. Train specifies whether the dataset contains a query-response training set. LLM-Gen indicates whether datasets are created by LLMs or humans. Rejection Rate denotes the average rejection rate across a fixed set of LLMs (see details in Appendix D ). Self-BLEU and Dist-2 (distinct-2gram) measure diversity. CoT indicates whether the dataset includes long chain-of-thought reasoning in responses. Over-Refusal Dataset. Prior datasets such as XSTest ( Röttger et al. 2024 ) and OKTest ( Shi et al. 2024 ) introduce safe prompts resembling harmful ones to evaluate false refusals, but are now too simple for SOTA models ( Cui et al. 2024 ) . WildGuardMix ( Han et al. 2024 ) provides broader safety evaluation across tasks but targets harmful content detection rather than refusal adjustment. PHTest ( An et al. 2024a ) and OR-Bench ( Cui et al. 2024 ) offer larger-scale pseudo-harmful prompts, yet rely on single-round LLM prompting (e.g., using Mixtral 8*7B ( Jiang et al. 2024 ) ) and lack diversity and quality control. Moreover, existing datasets contain only test sets with queries alone. In contrast, our FalseReject includes both queries and responses, supporting training methods like instruction tuning to actively calibrate refusal behavior. Table 1 summarizes key differences between FalseReject and previous datasets . Over-Refusal Mitigation. Previous methods mostly use training-free, inference-time mitigation approaches, orthogonal to our training-based approach. For instance, Self-CD ( Shi et al. 2024 ) contrasts outputs with varying safety emphasis during decoding to reduce sensitivity to harmless prompts. SCANS ( Cao et al. 2024 ) steers model activations adaptively based on safety classification, while single-vector ablation ( Wang et al. 2024a ; Arditi et al. 2024 ) removes specific false-refusal signals from the activation space. Unlike these methods applied to pre-aligned LLMs, our dataset targets earlier-stage alignment during post-training, directly influencing refusal behaviors. Other training-based approaches ( Zheng et al. 2024a ; Zhang et al. 2024d ) focus on preventing jailbreaks without addressing over-refusal explicitly. Jain et al. 2024 propose dynamic refusal control via special tokens but do not directly tackle false refusals. 3 FalseReject: A Large-Scale Over-Refusal Training and Evaluation Resource for LLMs We introduce FalseReject , a large-scale dataset for evaluating and calibrating LLMs’ over-refusal behavior. It contains 16k carefully curated prompts that appear harmful but are actually benign, covering 44 safety-related categories. 2 2 2 Following the taxonomy from Xie et al. 2024 ; see Figure 6 in Appendix B for the full list. The dataset includes a high-quality, human-annotated test set, FalseReject-Test ( 1.1k samples), and two training sets: FalseReject-Train-Instruct and FalseReject-Train-CoT , with 15k query-response pairs targeting non-reasoning and reasoning LLMs, respectively. Training on FalseReject helps LLMs better distinguish safe from unsafe contexts in controversial prompts, improving the tradeoff between safety and helpfulness. It effectively reduces unnecessary refusals while preserving general language capabilities. Example prompts and responses are shown in Figures 7 and 8 in the Appendix. Figure 2 shows our data generation pipeline, and we explain each stage in detail in the following sections. Figure 2: The overall pipeline for generating over-refusal queries in our FalseReject dataset. 3.1 Entity Graph Extraction Large-scale synthetic data generation with LLMs often results in repetitive content, reducing diversity ( Liu et al. 2024b ; Gandhi et al. 2024 ) . Inspired by recent advances in graph-informed synthetic data generation ( Yang et al. 2024b ; Wang et al. 2024b ; Zhang et al. 2024b ) , which have shown effectiveness in increasing diversity, we begin by extracting entity graphs from existing safety-related datasets. In particular, we use an LLM to identify and extract relevant entities from toxic prompts, focusing on people, locations, objects, and concepts associated with safety concerns. We collect multiple candidate lists of extracted entities and use an LLM-driven voting process to select the most representative one, forming a graph that encodes the relationships among these entities. Those graphs then serve as the foundation for the subsequent stage of our diverse over-refusal query generation process. We use Llama-3.1-405B for graph extraction, and the prompts are presented in Appendix E . 3.2 Iterative Adversarial Multi-Agent Interaction for Synthetic Queries Inspired by recent research in role-playing ( Chen et al. 2024 ; Tseng et al. 2024 ; Zhang et al. 2024c ) and multi-agent interactions among LLMs ( Du et al. 2023 ) , we propose an adversarial multi-agent debate framework composed of four main components: (1) a Generator, which creates seemingly harmful but benign prompts; (2) a Discriminator, tasked with evaluating whether these prompts are genuinely unsafe or merely appear unsafe; (3) a pool of LLMs performing refusal validation, which test each generated prompt and retain only those prompts refused by at least one LLM; and (4) an Orchestrator, responsible for determining whether these prompts constitute valid over-refusal cases, specifically evaluating if they are benign despite appearing sensitive and likely to trigger refusals from language models. In each iteration, the Generator either produces an initial query based on the entity graph or refines a previous one using feedback from the Discriminator. It actively tries to trigger refusals by generating prompts that seem increasingly unsafe but remain genuinely harmless. Meanwhile, the Discriminator tries to objectively evaluate prompts without being misled, identifying whether they are safe or unsafe. Empirically, we find that through this adversarial interaction, the Generator progressively improves its ability to create prompts that convincingly mimic harmful content yet remain ethically and factually benign. Each iteration involves inputting the generated prompt to the pool of validation LLMs, retaining the prompt only if at least one LLM refuses it. Subsequently, the Orchestrator conducts additional validation to confirm that this retained prompt appear sensitive enough to trigger refusals, yet it is objectively benign. This iterative adversarial procedure ensures the production of challenging synthetic queries that effectively simulate unsafe requests without actual harm. Figure 2 and Algorithm 1 illustrate our query generation process in detail. In practice, we suggest using strong LLMs such as Claude-3.5-Sonnet with different system prompts (provided in Appendix E ) to role-play the agents described in this section. Algorithm 1 Graph-Informed Adversarial Multi-Agent Interaction for Synthetic Over-Refusal Query Generation 1: Safety-related dataset \mathcal{D} with toxic prompts, LLM pool \{\mathcal{M}_{1},\dots,\mathcal{M}_{k}\} , maximum iterations N . 2: A set of over-refusal prompts \mathcal{P} that appear unsafe yet remain benign. 3: \mathcal{P}\leftarrow\emptyset \triangleright Initialize final prompt set 4: \mathcal{G}\leftarrow\textsc{ExtractGraph}(\mathcal{D}) \triangleright Entity-graph extraction for diversity 5: for t=1 to N do 6: \text{prompt}_{t}\leftarrow\textsc{Generator}\bigl(\mathcal{G},\{\text{feedback}_{i}\}_{i=1}^{t-1}\bigr) \triangleright Generate query 7: \text{disc\_decision}_{t},\;\text{disc\_feedback}_{t}\leftarrow\textsc{Discriminator}\bigl(\text{prompt}_{t}\bigr) 8: \text{refusals}\leftarrow\sum_{i=1}^{k}\textsc{TestRefusal}\bigl(\mathcal{M}_{i},\;\text{prompt}_{t}\bigr) \triangleright Check refusal across LLMs 9: if \text{refusals}>0 then 10: \text{orch\_decision}_{t}\leftarrow\textsc{Orchestrator}\bigl(\text{prompt}_{t},\;\text{disc\_decision}_{t},\;\text{disc\_feedback}_{t}\bigr) 11: if \text{orch\_decision}_{t}=\text{``valid''} then 12: \mathcal{P}\leftarrow\mathcal{P}\cup\{\text{prompt}_{t}\} \triangleright Add prompt to final set 13: break \triangleright End loop upon successful acceptance 14: end if 15: end if 16: Update \bigl(\text{feedback}_{t},\;\text{prompt}_{t},\;\text{disc\_decision}_{t},\;\text{disc\_feedback}_{t}\bigr) 17: end for 18: return \mathcal{P} 3.3 Quality Control and Human Annotation To further ensure high-quality queries, we implement several filtering and augmentation procedures. First, we remove duplicated or highly similar prompts to maximize dataset diversity. Next, we balance the topic distribution to comprehensively cover all 44 safety-related categories, ensuring broad topical representation. For additional quality assurance in constructing a reliable test set, we recruit human annotators to evaluate query sensitivity and determine whether each query needs a direct refusal or could instead be answered safely within an appropriate context. We select queries identified by annotators as seemingly sensitive yet safely answerable when sufficient context is provided. These queries form the basis of our test dataset. Detailed annotation procedures and the complete annotation interface used by annotators are described in Appendix F . Overall, we obtain 15k queries for the training set, for which we describe the response generation process in the following section. Our finalized test set, named FalseReject-Test , comprises 1.1k data points. Figure 8 presents examples of queries in FalseReject-Test and comparisons with previous datasets. 3.4 Context-Aware Safety Response Generation via Structured Reasoning Traces One significant reason behind the problem of LLM over-refusal is ambiguity , an inherent characteristic of natural language ( Piantadosi et al. 2012 ; Liu et al. 2023 ) . Many queries have multiple possible interpretations, with some being safe and others potentially unsafe from the perspective of LLMs. Prior work ( An et al. 2024b ) identified that such ambiguous inputs can cause LLMs to refuse responses, categorizing these cases as controversial . Building on this observation, we argue that many of these controversial queries should not be directly refused, as doing so may result in the loss of information that could be helpful when the user’s intent is benign. Namely, responses should be context-aware , following the user’s instructions in safe contexts while carefully avoiding the generation of unsafe content. Motivated by this insight, we construct an instruction-tuning dataset designed to train LLMs to explicitly distinguish between these contexts , enabling them to provide useful information wherever possible while maintaining responsible behavior. Building on queries obtained and following Brahman et al. 2024 , we prompt LLMs with a structured reasoning instruction to generate responses that explicitly differentiate safe interpretations from potentially unsafe ones. Specifically, we use a standard LLM 3 3 3 We recommend using Claude-3.5-Sonnet or a model with comparable capabilities. for regular response generation, creating the FalseReject-Train-Instruct dataset for models without inherent reasoning capabilities, and DeepSeek-R1 for generating reasoning-intensive responses, forming the FalseReject-Train-CoT dataset for reasoning models. The reasoning generation includes a monologue-style thinking process encapsulated within special identifier tokens, followed by a final solution presented clearly to the user. To guide the models effectively, we provide the name of the subcategory , its definition , and the expected response format . Models are instructed to follow a structured response consisting of: (1) Acknowledgment and Differentiation of Multiple Contexts : The response begins by explicitly identifying both safe and unsafe contexts in the query. (2) Detailed Explanation of the Safe Context : The model provides a comprehensive and informative answer addressing the safe interpretation of the query. (3) Clarification and Guidance on Potentially Unsafe Contexts : If certain aspects of the query could involve risk, the model explicitly explains concerns and encourages users to clarify their intent or seek guidance from qualified professionals. (4) Closing Statement : The response concludes by summarizing the safe information provided and emphasizing caution and careful consideration in unsafe situations. The generated responses maintain coherent, conversational flow, ensuring that they avoid directly refusing controversial queries. Figure 7 in the Appendix provides an example from our instruction-tuning dataset, and the prompts used for response generation are detailed in Appendix E . 4 Benchmarking LLMs with FalseReject Figure 3: Benchmarking results on the FalseReject-Test dataset, comparing Compliance Rate and USR metrics across various language models. Closed-source models are indicated with dark green labels, while open-source models are shown in black. Reasoning-specific models ( o1 , Deepseek-R1 , and QwQ ) are additionally marked with a star ( \star ). Models to Test. We comprehensively evaluate 29 SOTA LLMs from diverse families on FalseReject-Test, covering both closed-source and open-source implementations. The evaluated models include the GPT series (GPT-4.5, GPT-4o, o1) ( Achiam et al. 2023 ; Hurst et al. 2024 ; Jaech et al. 2024 ) , the Claude series (Claude-3.7-sonnet, Claude-3.5-sonnet, Claude-3.5-Haiku, Claude-3.0-opus) ( Anthropic 2025 ) , the Gemini series (Gemini-2.5-Pro, Gemini-2.0-Pro) ( Team et al. 2023 ) , the Llama-3 series (1B, 3B, 8B, 70B, 405B) ( Grattafiori et al. 2024 ) , and the Gemma-3 series (1B, 4B, 27B) ( Team et al. 2024 ) . We also include Mistral-7B Instruct v0.2 ( Jiang et al. 2023 ) , Cohere Command-R+ ( Cohere 2025 ) , the Qwen-2.5 series (0.5B, 1.5B, 7B, 14B, 32B) ( Yang et al. 2024a ) , QwQ-32B-Preview ( Qwen Team 2024 ) , the Phi series (Phi-4 ( Abdin et al. 2024 ) , Phi-4-mini ( Abouelenin et al. 2025 ) ), and the DeepSeek series (DeepSeek-V3 ( Liu et al. 2024a ) , DeepSeek-R1 ( Guo et al. 2025 ) ). Evaluation and Metric. Prior works ( Cui et al. 2024 ; Brahman et al. 2024 ; Röttger et al. 2024 ) often use keyword matching (e.g., “I’m sorry”) to classify LLM responses as compliant or refusal, which can be imprecise and overlook subtle refusals. Motivated by recent findings that LLM-as-a-judge methods ( Cui et al. 2024 ; Röttger et al. 2024 ; Han et al. 2024 ) offer more reliable evaluations, we use Claude-3.5-Sonnet for response classification. Following these works, we report the Compliance Rate , the proportion of compliant responses (1 - rejection rate). To jointly assess safety and helpfulness, we introduce a three-class metric, Useful Safety Rate (USR) , which distinguishes Direct Refusal, Safe Partial Compliance, and Full Compliance. Formally: For toxic prompts: \text{USR}_{\text{Toxic}}=\frac{\#(\text{Direct Refusal})+\#(\text{Safe Partial Compliance})}{\#(\text{Total Toxic Prompts})} For benign prompts: \text{USR}_{\text{Benign}}=\frac{\#(\text{Full Compliance})+\#(\text{Safe Partial Compliance})}{\#(\text{Total Benign Prompts})} Safe Partial Compliance refers to responses that recognize safety concerns and avoid harmful content while still constructively engaging with the prompt (examples in Appendix E ). A higher \text{USR}_{\text{Toxic}} reflects better safety , and a higher \text{USR}_{\text{Benign}} indicates improved helpfulness with fewer refusals on benign prompts. USR enhances binary metrics by explicitly separating partial compliance from outright refusals. Following Cui et al. 2024 , we evaluate over-refusal on benign prompts. The three-class classification prompt is in Appendix E . 4.1 Results and Findings We present compliance rates and \text{USR}_{\text{Benign}} of various LLMs on our FalseReject-Test in Figure 3 . Our main findings are detailed as follows: FalseReject Remains Challenging: Significant Over-Refusal by SOTA LLMs. The results demonstrate that even SOTA models still struggle significantly with over-refusal. Most models show compliance rates and \text{USR}_{\text{Benign}} far from perfect (i.e., approaching 100%). For instance, widely-used models such as GPT-4.5 and Claude-3.5-Sonnet have compliance rates below 50%, emphasizing the persistent challenge in accurately balancing safety with helpfulness. Reasoning Models Show Inconsistent Over-Refusal Behavior. Comparing reasoning-oriented models such as QwQ-32B-Preview, DeepSeek-R1, and o1, we observe inconsistent behavior. While DeepSeek-R1 achieves the highest compliance rate (87.53%) and nearly perfect USR (99.66%), both QwQ and o1 exhibit substantially lower compliance rates, significantly underperforming compared to the average of non-reasoning models. This suggests variability in how reasoning-oriented post-training handle alignment related to safe and helpful responses. Model Families Exhibit Distinct Refusal Patterns. According to the definition of compliance rate and USR , the gap between these two metrics approximates the percentage of responses exhibiting safe partial compliance. Different model series display significant different refusal patterns. For example, the Phi-4 series models have a notable gap (approximately 30-40%), suggesting frequent partial compliance. Conversely, GPT series models such as GPT-4o demonstrate much narrower gaps (less than 15 %), indicating they either directly refuse or fully comply in most cases, highlighting different strategies employed during their respective safety alignment processes. Model Size is Not a Noticeable Factor. Analyzing models across varying scales within the same family, such as the Llama-3 series (1B to 405B) and Qwen-2.5 series (0.5B to 32B), we find no consistent relationship between model size and refusal metrics. Smaller models like Llama-3.2-1B (compliance rate: 78.43%, USR : 86.60%) outperform larger counterparts like Llama-3-1-405B (compliance rate: 56.28%, USR : 74.56%), indicating that model capacity alone does not significantly influence over-refusal tendencies. General Language Ability Does Not Strongly Predict Over-Refusal Behavior. Surprisingly, high-performing general-purpose LLMs such as GPT-4.5 and O1 exhibit lower compliance rate and USR than models with generally weaker general language abilities, such as Llama-3.2-1B and Phi4-mini. This suggests that proficiency in general language tasks does not necessarily correlate with improved over-refusal behavior, underscoring that alignment strategies specifically targeting refusal behavior require distinct optimization objectives. Open-Source Models Can Potentially Outperform Closed-Source Counterparts on Over-Refusal Metrics. Interestingly, some open-source models demonstrate notably high performance on our over-refusal metrics, potentially outperforming closed-source models. For instance, open-source models such as Mistral-7B (compliance rate: 82.14%, USR : 99.49%) and DeepSeek-R1 (compliance rate: 87.53%, USR : 99.66%) show strong results compared to closed-source models like GPT-4.5 and the Claude-3 series. This highlights the growing capability of open-source models and suggests that competitive alignment performance is achievable in open communities. We conduct a more in-depth analysis of how models from different families overlap in their refusals on data points in the FalseReject-Test in Appendix C . 5 Finetuning To Mitigate Over-refusal with FalseReject Training Data. Following Brahman et al. 2024 ; Zhang et al. 2024a , we use a general-purpose instruction tuning dataset to balance safety and helpfulness. We select utility data from two sources: Open-Thoughts-114k ( Team 2025 ) , a synthetic reasoning dataset generated by Deepseek-R1 ( Guo et al. 2025 ) with 114k CoT examples for training reasoning models, and Tulu-3-SFT-mixture ( Lambert et al. 2024 ) , a 940k-instance dataset spanning diverse tasks for training non-reasoning models. We follow a 90:10 utility-to-safety data ratio as in Zhang et al. 2024a . Due to compute limits, we sample 12,000 pairs from Open-Thoughts or Tulu-3, and 1,300 from FalseReject-Train-CoT or FalseReject-Train-Instruct, depending on whether the target is a reasoning or non-reasoning model. Models. We train several LLMs of varying sizes, including Llama-3.2-1B, Llama-3-8B ( Grattafiori et al. 2024 ) , Qwen-2.5-0.5B, Qwen-2.5-7B ( Yang et al. 2024a ) , and Gemma-2-2B ( Team et al. 2024 ) . Following prior work ( Brahman et al. 2024 ; Zhang et al. 2024a ) , we conduct SFT on the base pretrained models rather than their instruction-tuned variants, to avoid confounding from built-in safety tuning. To assess the impact of our training strategy, we compare models fine-tuned on combined utility and safety data against baselines trained only on utility data. Evaluation Setup. In addition to FalseReject-Test, we also evaluate over-refusal behavior using OR-Bench-Hard-1K ( Cui et al. 2024 ) , a subset filtered from OR-Bench based on LLM rejection rates. To evaluate model safety, we follow Zhang et al. 2024a and use datasets including AdvBench ( Zou et al. 2023 ) , MaliciousInstructions ( Bianchi et al. 2023 ) , Sorry-Bench ( Xie et al. 2024 ) , and StrongREJECT ( Souly et al. 2024 ) . General language abilities and knowledge understanding are assessed with the widely-used GSM8K ( Cobbe et al. 2021 ) for grade-level math reasoning and MMLU ( Hendrycks et al. 2020 ) for broader language comprehension. Additional implementation details are provided in Appendix A . Model Tuning General Safety Over-Refusal MMLU-0 GSM8K AB MI SR SB Or-Bench FalseReject Qwen-2.5 0.5B Tulu-3 (baseline) 40.30 39.42 99.23% 94.00% 92.01% 90.44% 56.48% 70.09% Tulu-3 + FalseReject-Train-Instruct 38.20 37.53 99.62% 93.00% 95.53% 92.89% 69.60% 95.62% OpenThoughts (baseline) 36.56 33.36 66.92% 56.00% 82.11% 78.22% 76.80% 71.44% OpenThoughts + FalseReject-Train-CoT 36.00 33.18 97.69% 100.0% 96.49% 97.11% 99.92% 99.58% Llama-3.2 1B Tulu-3 (baseline) 30.30 23.35 99.23% 100.0% 94.89% 94.44% 56.48% 66.39% Tulu-3 + FalseReject-Train-Instruct 32.30 22.59 100.0% 100.0% 97.44% 94.00% 69.60% 97.14% OpenThoughts (baseline) 34.80 24.10 43.27% 42.00% 67.41% 61.11% 87.34% 87.95% OpenThoughts + FalseReject-Train-CoT 32.60 26.88 99.23% 99.00% 96.81% 97.11% 99.85% 100.0% Gemma-2 2B Tulu-3 (baseline) 39.60 37.98 99.81% 100.0% 98.08% 94.00% 54.44% 69.59% Tulu-3 + FalseReject-Train-Instruct 40.66 37.60 100.0% 100.0% 99.04% 96.22% 73.69% 98.65% OpenThoughts (baseline) 49.23 42.76 29.23% 19.00% 49.20% 52.00% 96.06% 94.44% OpenThoughts + FalseReject-Train-CoT 49.35 46.47 100.0% 100.0% 98.40% 97.78% 100.0% 99.92% Qwen-2.5 7B Tulu-3 (baseline) 68.50 68.61 100.0% 100.0% 99.36% 91.78% 53.30% 65.54% Tulu-3 + FalseReject-Train-Instruct 68.60 72.63 100.0% 100.0% 100.0% 94.67% 77.48% 99.24% OpenThoughts (baseline) 76.10 91.88 32.31% 20.00% 44.73% 44.44% 97.80% 96.71% OpenThoughts + FalseReject-Train-CoT 75.10 90.90 100.0% 100.0% 99.68% 96.67% 99.85% 100.0% Llama-3 8B Tulu-3 (baseline) 52.20 51.33 100.0% 99.0% 99.36% 94.67% 44.58% 57.37% Tulu-3 + FalseReject-Train-Instruct 54.00 51.48 100.0% 100.0% 99.68% 95.56% 64.67% 98.82% OpenThoughts (baseline) 66.60 81.35 17.12% 17.00% 38.34% 32.89% 98.33% 97.30% OpenThoughts + FalseReject-Train-CoT 66.67 82.54 100.0% 99.00% 100.0% 97.78% 100.0% 100.0% Table 2: Training with FalseReject effectively mitigates over-refusal in non-reasoning models and improves safety in reasoning models. We report USR scores across six sources of safety and over-refusal prompts: AdvBench (AB), MaliciousInstructions (MI), StrongReject (SR), Sorry-Bench (SB), and Or-Bench-1k-Hard (Or-Bench). Results are shown for models trained with our dataset and baseline methods. General language ability evaluation scores are also included. Higher scores indicate better performance across all metrics. We bold the better results for over-refusal evaluation. 5.1 Results and Findings We report \text{USR}_{\text{Benign}} for evaluating over-refusal, \text{USR}_{\text{Toxic}} for evaluating safety, and accuracy on general language utility datasets in Table 2 . Our main findings are summarized below. SFT with FalseReject-Train-Instruct effectively mitigates over-refusal in non-reasoning LLMs. We find that incorporating FalseReject-Train-Instruct into Tulu-3 and finetuning LLMs of different sizes consistently leads to significant improvements in USR compared to baselines. These gains are achieved with minimal loss in general language ability and even slight improvements in safety metrics where baseline models already perform well. This demonstrates that our training data is effective for balancing helpfulness and safety during the post-training phase of non-reasoning LLMs. As a case study shown in Figure 1 , models fine-tuned with our dataset can reliably distinguish between safe and unsafe contexts, offering helpful information where appropriate and withholding it when necessary. SFT with FalseReject-Train-CoT substantially improves safety in reasoning LLMs. We observe that reasoning models trained solely on general CoT datasets achieve high compliance rates in over-refusal evaluations and perform well on general language utility tasks, but they struggle severely in safety evaluations. This aligns with recent findings that open-source reasoning models often exhibit significant safety vulnerabilities ( Zhou et al. 2025 ) . In contrast, when models are trained with a mixture that includes FalseReject-Train-CoT, their safety performance improves significantly, even surpassing that of non-reasoning models, while maintaining near-perfect results on over-refusal benchmarks and preserving general utility. These results highlight that FalseReject-Train-CoT is a valuable resource for post-training calibration of reasoning models. 5.2 In-depth Analysis Recent work ( Qi et al. 2024 ) identified an issue termed shallow safety alignment , where alignment processes mainly adjust a model’s generative distribution primarily over the initial few tokens, making safety-aligned models vulnerable to attacks such as the prefilling attack ( Zou et al. 2023 ) . To verify the alignment depth of models trained with our dataset, we adopt the same approach proposed by Qi et al. 2024 , examining token-wise distribution differences between the aligned model \pi_{\text{aligned}} and its base model \pi_{\text{base}} . Specifically, we select 1,000 instruction-response pairs (x,y) from our FalseReject dataset, where all responses are classified as refusals. For each token position k within the response y , we compute the token-wise KL divergence as D_{\text{KL}}\left(\pi_{\text{aligned}}(\cdot\mid x,y_{