Do Language Models Know When They’ll Refuse? Probing Introspective Awareness of Safety Boundaries

Tanay Gondil Purdue University [email protected] · 2026

Read Do Language Models Know When They’ll Refuse? Probing Introspective Awareness of Safety Boundaries by Tanay Gondil Purdue University [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00228.txt.

Introduction

Tanay Gondil asks whether language models can predict their own refusals before answering, testing models forecasting refusal before responding. Across 3754 points and 300 requests, four models show introspection, with d-prime values between 2.4 and 3.5, though sensitivity falls at boundaries. Claude Sonnet 4.5 leads its family at about 96 percent against about 93 percent for Sonnet 4, while GPT-5.2 trails at about 89 percent and Llama at 80 percent with refusal bias and poor calibration. Weapons queries prove hardest, and high-confidence predictions reach about 98 percent for calibrated models, enabling confidence-based routing.

Catalog record

CREATORTanay Gondil Purdue University [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23420
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00228

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Language Models Know When They’ll Refuse? Probing Introspective Awareness of Safety Boundaries Tanay Gondil Purdue University [email protected] Abstract Large language models are trained to refuse harmful requests, but can they accurately predict when they will refuse before responding? We investigate this question through a systematic study where models first predict their refusal behavior, then respond in a fresh context. Across 3,754 datapoints spanning 300 requests, we evaluate four frontier models: Claude Sonnet 4, Claude Sonnet 4.5, GPT-5.2, and Llama 3.1 405B. Using signal detection theory (SDT), we find that all models exhibit high introspective sensitivity ( d ′ d^{\prime} =2.4–3.5), but sensitivity drops substantially at safety boundaries. We observe generational improvement within Claude (Sonnet 4.5: 95.7% accuracy [95% CI: 94.4–96.9], ECE=0.017 vs Sonnet 4: 93.0% [91.3–94.7], ECE=0.048), while GPT-5.2 shows lower accuracy (88.9% [86.8–90.9]) with more variable behavior. Notably, the open-source Llama 405B achieves high sensitivity ( d ′ d^{\prime} =3.29) but exhibits strong refusal bias (criterion= − - 0.86) and poor calibration (ECE=0.216), resulting in only 80.0% accuracy despite good discrimination. Topic-wise analysis reveals weapons-related queries are consistently hardest for introspection. Critically, confidence scores provide actionable signal: restricting to high-confidence predictions (conf ≥ \geq 5) yields 98.3% accuracy for Sonnet 4.5 with 85% coverage, enabling practical confidence-based routing for safety-critical deployments. 1 Introduction Modern large language models undergo extensive safety training to refuse harmful requests while remaining helpful for benign ones ( 9 ; 2 ) . While substantial research has examined when models refuse, how to elicit or bypass refusals, and whether refusals are robust to adversarial attacks ( 15 ; 12 ) , a more fundamental question remains underexplored: Do language models know when they will refuse? This question probes the introspective awareness of LLMs regarding their own safety boundaries. When presented with a potentially harmful request, can a model accurately predict whether it will refuse before actually generating a response? The answer has both theoretical and practical significance. Theoretical significance. If models can accurately predict their refusal behavior, this suggests that safety training creates explicit, accessible representations of harmfulness that the model can query introspectively. We formalize this using signal detection theory (SDT), treating refusal prediction as a detection task where the model must discriminate between requests it will refuse versus comply with. This framework yields sensitivity ( d^{\prime} ) and bias (criterion) metrics that provide mechanistic insight into how introspective access varies across request types. Practical significance. Accurate self-prediction enables confidence-based routing : models can signal uncertainty about their safety decisions, allowing systems to route ambiguous cases for human review. We demonstrate that this is achievable in practice: high-confidence predictions from well-calibrated models achieve near-perfect accuracy. We systematically investigate LLM introspective awareness of refusal behavior. Our protocol is simple: (1) present a request and ask the model to predict whether it will refuse, (2) in a fresh context, present the same request and observe actual behavior, (3) compare prediction to reality. We apply this across 300 requests spanning 10 sensitive topics and 5 harm levels, with paraphrases to test consistency. Our main contributions are: 1. SDT framework for introspection : We apply signal detection theory to quantify introspective sensitivity, finding d^{\prime} =2.4–3.5 overall but substantial degradation at boundaries ( d^{\prime} drops 40–75%). 2. Mechanistic insights : Topic-wise analysis reveals weapons-related queries are hardest for introspection; error analysis shows Level 4 (“likely harmful”) requests, not borderline Level 3, drive most errors. 3. Actionable recommendations : We demonstrate that confidence-based routing is viable: conf \geq 5 predictions achieve 98.3% accuracy at 85% coverage for well-calibrated models. 4. Cross-family comparison : We compare closed-source (Claude, GPT) and open-source (Llama) models, finding that Llama 405B achieves high discrimination ( d^{\prime} =3.29) but exhibits strong refusal bias and poor calibration, highlighting that introspective accuracy depends on both sensitivity and bias. 2 Related Work LLM Safety and Refusal Behavior. Safety training has evolved from keyword filtering to RLHF ( 9 ) and constitutional AI ( 2 ) . Work has studied refusal robustness, including adversarial attacks ( 15 ; 12 ) and red-teaming ( 10 ; 5 ) . We study the model’s self-knowledge of its refusal behavior rather than the behavior itself. Introspection and Self-Knowledge in LLMs. 8 found models can often identify when they know factual information, though with failures. 14 examined whether models know what they don’t know with mixed results. Recent work has intensified: 3 showed LLMs can learn about themselves through introspection, achieving calibrated self-predictions. Concurrent work on emergent introspective awareness ( 1 ) uses mechanistic methods to study whether models can access their internal states. We extend introspection analysis to the safety domain using a rigorous SDT framework, focusing on behavioral prediction rather than factual knowledge. Signal Detection Theory. SDT provides a principled framework for analyzing detection performance, separating sensitivity ( d^{\prime} ) from response bias ( 6 ) . While SDT has been applied to human metacognition ( 4 ) , its application to LLM introspection is novel. Calibration. Confidence calibration has been studied in neural networks ( 7 ) and LLMs ( 13 ; 11 ) . Our finding that calibration quality varies dramatically across models (ECE=0.017 for Sonnet 4.5 vs 0.216 for Llama) extends this work to safety decisions and demonstrates practical implications for routing. 3 Methodology 3.1 Experimental Protocol Our protocol consists of three phases: Phase 1 (Prediction): We present the model with a request and ask it to predict: (1) Will it refuse? (yes/no), (2) Confidence (1–5), (3) Harm rating (1–5). Phase 2 (Actual Response): In a fresh context with no memory of Phase 1, we present the same request and collect the actual response. Phase 3 (Classification): We classify the response as: full_refusal, partial_refusal, partial_compliance, or full_compliance. For binary analysis, we treat refusal categories as “refused” and compliance categories as “complied.” 3.2 Dataset We construct 300 base requests spanning 10 sensitive topics (weapons, drugs, hacking, self-harm, hate speech, fraud, privacy, illegal activities, manipulation, violence) across 5 harm levels: • Level 1 (5 per topic): Clearly safe educational queries • Level 2 (5 per topic): Edgy but benign requests • Level 3 (10 per topic): Borderline requests • Level 4 (5 per topic): Likely harmful requests • Level 5 (5 per topic): Clearly harmful, from AdvBench ( 15 ) Each request has 2 paraphrases (900 total datapoints per model). 3.3 Empirical Ground Truth We use empirical refusal rates as ground truth rather than assigned harm labels. For each request, we compute the fraction of times the model refused across variants and categorize: • Safe (0–20% refusal): Model consistently complies • Leaning safe (20–40%): Model usually complies • Leaning harmful (60–80%): Model usually refuses • Harmful (80–100%): Model consistently refuses This measures introspection against actual behavior, not preconceptions. 3.4 Signal Detection Theory Framework We formalize introspection as a signal detection task. Let the “signal” be that the model will refuse (actual_refused=True) and the “response” be that the model predicts refusal (predicted_refuse=True). This yields four outcomes: • Hit : Predicted refuse AND actually refused • Miss : Predicted comply AND actually refused • False Alarm : Predicted refuse AND actually complied • Correct Rejection : Predicted comply AND actually complied We compute sensitivity and criterion as: \displaystyle d^{\prime} \displaystyle=z(\text{hit rate})-z(\text{false alarm

Read the complete text