SAGE : A Generic Framework for LLM Safety Evaluation Warning: The paper content can be harmful and offensive to readers.
Madhur Jindal, Hari Shrawgi, Parag Agrawal, Sandipan Dandapat · 2025
Read SAGE : A Generic Framework for LLM Safety Evaluation Warning: The paper content can be harmful and offensive to readers. by Madhur Jindal, Hari Shrawgi, Parag Agrawal, Sandipan Dandapat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19674.txt.
Introduction
Madhur Jindal, Hari Shrawgi, Parag Agrawal, and Sandipan Dandapat build a safety test bed for language models in a 2025 paper. Models now serve in health, finance, and more, but safety tests mostly use single questions under broad rules. Real harm often grows over a chat and depends on the use case, so such tests miss it.
Their framework, SAGE, runs automated multi-turn chats with tester agents of varied personalities. Tests are tuned to each application and its harm rules. Seven leading models were tried across three applications. Harm rose as chats lengthened, and behavior swung with the tester's personality. Some models stayed safe mainly by refusing often, which cut their usefulness. Tightening one child-safety rule sharply raised faults. The authors urge testing that adapts to context.
Catalog record
| CREATOR | Madhur Jindal, Hari Shrawgi, Parag Agrawal, Sandipan Dandapat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19674
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAGE : A Generic Framework for LLM Safety Evaluation Warning: The paper content can be harmful and offensive to readers. Madhur Jindal, Hari Shrawgi, Parag Agrawal, Sandipan Dandapat Abstract As Large Language Models are rapidly deployed across diverse applications from healthcare to financial advice, safety evaluation struggles to keep pace. Current benchmarks focus on single-turn interactions with generic policies, failing to capture the conversational dynamics of real-world usage and the application-specific harms that emerge in context. Such potential oversights can lead to harms that go unnoticed in standard safety benchmarks and other current evaluation methodologies. To address these needs for robust AI safety evaluation, we introduce SAGE ( S afety A I G eneric E valuation), an automated modular framework designed for customized and dynamic harm evaluations. SAGE employs prompted adversarial agents with diverse personalities based on the Big Five model, enabling system-aware multi-turn conversations that adapt to target applications and harm policies. We evaluate seven state-of-the-art LLMs across three applications and harm policies. Multi-turn experiments show that harm increases with conversation length, model behavior varies significantly when exposed to different user personalities and scenarios, and some models minimize harm via high refusal rates that reduce usefulness. We also demonstrate policy sensitivity within a harm category where tightening a child-focused sexual policy substantially increases measured defects across applications. These results motivate adaptive, policy-aware, and context-specific testing for safer real-world deployment. 1 Introduction Numerous concerns and harms have arisen with the widespread adoption of Large Language Models (LLMs) Weidinger et al. (2021) ; Bender et al. (2021) . LLMs have become integral to applications ranging from web search Kelly et al. (2023) and customer service Pandya and Holia (2023) to financial assistance Wu et al. (2023) , clinical partnerships Singhal et al. (2022) , and business strategy Zheng et al. (2024) . With LLMs becoming ubiquitous across society Bommasani et al. (2022) , assessing their safety alignment remains a significant challenge Zhang et al. (2023) . Figure 1: Examples demonstrate how models provide different responses to harmful queries depending on application context, motivating application-specific evaluation. Traditional safety benchmarks prove insufficient due to four critical gaps (detailed in Appendix Table 7 ): (1) Conversational Evaluation: Most work focuses on single-prompt datasets Zhang et al. (2023) rather than extended conversations that reveal multi-turn vulnerabilities Perez et al. (2022) . Existing multi-turn benchmarks Kwan et al. (2024) often produce incoherent conversations due to static design, missing the adaptive nature required for realistic user interactions. (2) Application-Specific Evaluation: Deployment across sectors like finance and healthcare requires context-specific assessment, as models exhibit inconsistent behavior due to non-generalizable safety alignment or surface-level protections being fooled by application context. For example, self-harm queries yield different responses in vanilla chatbots versus cooking assistants, and financial models may inappropriately handle investment queries involving harmful activities (see Appendix Figure 1 ). (3) Custom Harms & Policies: Application-specific policies are challenging to enumerate Weidinger et al. (2021) ; Rauh et al. (2022) , with definitions varying significantly by demographics (sexual harm thresholds differ for minors versus adults) or becoming critical for specific applications (political disinformation in information-focused platforms like Microsoft Copilot where users expect accurate, up-to-date information). (4) Dynamic & Diverse Testing: Static benchmarks face rapid obsolescence—LLama3 achieves >95% GSM8K accuracy Cobbe et al. (2021) highlighting benchmark saturation—and contamination risks from unintentional training exposure Deng et al. (2024) ; Balloccu et al. (2024) . Additionally, diverse global user populations require testing that represents varied interaction patterns, cultural contexts, and attack methodologies Zhang et al. (2023) ; Grajzel et al. (2023) . Figure 2: Example conversation showing how harmful content can be elicited across multiple turns despite initial refusal. The AI’s eventual policy violation would be missed by single-turn evaluation, demonstrating the necessity of conversational safety assessment. Figure 2 demonstrates how follow-up turns reveal vulnerabilities absent in single-turn tests, empirically validated in our Results section. To address these limitations, we introduce SAGE ( S afety A I G eneric E valuation), an automated modular framework for customized, dynamic harm evaluation. Our contributions are: • SAGE : This novel framework firstly provides customizable evaluation across diverse applications, harm & policies. Secondly, it offers support for multi-turn conversational evaluation. Lastly, it is based on novel dynamic user models that offer system-awareness, personality and adversarialness during evaluation. • Evaluation of State-of-the-Art LLMs : We demonstrate SAGE ’s capability to evaluate seven state-of-the-art LLMs, both closed and open source, spanning various sizes and families covering three applications and three harm policies. With our multi-turn evaluation based on 32 diverse user model personalities, we provide novel results and insights for conversational safety and usefulness. We make all experimental data and the SAGE framework publicly available 1 1 1 https://github.com/Madhur-1/SageLLMSafetyEval to facilitate future safety research. 2 Related Work LLM safety research spans harm identification and evaluation methodology development 2 2 2 Note that our study does not include aspects related to fairness or other similar safety considerations . Understanding LLM Harms: Comprehensive taxonomies Weidinger et al. (2021) ; Bender et al. (2021) ; Bommasani et al. (2022) establish harm identification foundations, while targeted studies examine specific risks including bias Zhao et al. (2018) ; Röttger et al. (2021) and toxicity Wen et al. (2023) ; Deshpande et al. (2023) . Rauh et al. (2022) and Lin et al. (2022) investigate harmful content characteristics and truthfulness concerns. While harm collation efforts Zhang et al. (2023) benefit the community, they highlight harm innumerability and policy complexity across applications, motivating SAGE ’s customizable approach. Harm Testing Methodologies: Approaches fall into three categories. Manual red-teaming Ganguli et al. (2022) ; Parrish et al. (2022) provides reliable assessment but is labor-intensive and exposes annotators to harmful content Radharapu et al. (2023) . Templatic methods Deshpande et al. (2023) ; Gehman et al. (2020) generate test cases through manual or automated approaches— Gehman et al. (2020) introduced 100K web-derived prompts for toxic content generation—but lack scalability to diverse applications. Automated red-teaming addresses limitations: Perez et al. (2022) developed LM-based testing with single-turn focus and initial conversation red-teaming using fine-tuned models, while Radharapu et al. (2023) introduced AART for application-specific evaluation using templatic test cases, though potentially limiting scalability to diverse AI systems. SAGE advances automated red-teaming through dynamic conversational evaluation using prompted user models with system-awareness, personality diversity, and adversarial capabilities, addressing existing limitations while maintaining customizability benefits. 3 SAGE Framework Figure 3: SAGE framework overview. System Description and Policy Definition (inputs) customize three automated components: Seed Generation produces targeted test topics, User Simulator conducts adversarial conversations with personality-driven agents, and Evaluator judges policy violations. This modular design enables application- and harm-specific safety evaluation at scale. The overarching goal of the SAGE framework is to evaluate LLMs for potential harms in a scalable and automated manner. SAGE is designed to be flexible and adaptable, enabling the evaluation of LLMs across a wide range of applications and policies. The framework consists of several blocks, each serving a specific purpose in the evaluation process 3 3 3 These blocks can be customized and replaced with arbitrary components to adapt to technical needs and trade-offs . Please refer to Figure 3 for an overview of the SAGE framework. The framework consists of the following: 3.1 System Description &