RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs’ Contextual Sensitivity

Jisu Shin, Hoyun Song, Juhyun Oh, Changgeon Ko, Eunsu Kim, Chani Jung & Alice Oh · 2025

Read RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs’ Contextual Sensitivity by Jisu Shin, Hoyun Song, Juhyun Oh, Changgeon Ko, Eunsu Kim, Chani Jung & Alice Oh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25897.txt.

Introduction

Jisu Shin and co-authors build in this 2025 paper a benchmark that puts language models in social binds. Role conflicts pit the duties of one role against another: the loyal friend against the honest witness. Their set holds over thirteen thousand scenarios across sixty-five roles in five life domains, each tuned by how urgent the clashing claims are.

Urgency gives an objective yardstick in a subjective field. Ten leading models mostly failed it: instead of reading the situation, they fell back on baked-in preferences for certain roles. The authors present the gap as a measure of contextual sensitivity that training has yet to teach. Chatbots giving life advice face the same binds.

Catalog record

CREATORJisu Shin, Hoyun Song, Juhyun Oh, Changgeon Ko, Eunsu Kim, Chani Jung & Alice Oh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs’ Contextual Sensitivity Jisu Shin, Hoyun Song, Juhyun Oh, Changgeon Ko, Eunsu Kim, Chani Jung & Alice Oh Abstract Humans often encounter role conflicts—social dilemmas where the expectations of multiple roles clash and cannot be simultaneously fulfilled. As large language models (LLMs) become increasingly influential in human decision-making, understanding how they behave in complex social situations is essential. While previous research has evaluated LLMs’ social abilities in contexts with predefined correct answers, role conflicts represent inherently ambiguous social dilemmas that require contextual sensitivity: the ability to recognize and appropriately weigh situational cues that can fundamentally alter decision priorities. To address this gap, we introduce RoleConflictBench , a novel benchmark designed to evaluate LLMs’ contextual sensitivity in complex social dilemmas. Our benchmark employs a three-stage pipeline to generate over 13K realistic role conflict scenarios across 65 roles, systematically varying their associated expectations (i.e., their responsibilities and obligations) and situational urgency levels. By analyzing model choices across 10 different LLMs, we find that while LLMs show some capacity to respond to these contextual cues, this sensitivity is insufficient. Instead, their decisions are predominantly governed by a powerful, inherent bias related to social roles rather than situational information. Our analysis quantifies these biases, revealing a dominant preference for roles within the Family and Occupation domains, as well as a clear prioritization of male roles and Abrahamic religions across most evaluatee models. Code & Dataset: https://github.com/ddindidu/RoleConflictBench 1 Introduction Figure 1: Conceptual illustration of RoleConflictBench . We generate distinct expectations for two competing social roles and synthesize them into a story depicting an individual’s role conflict. Our benchmark is designed to evaluate how decisions change depending on the situation. Imagine a researcher working against a crucial paper submission deadline when they receive an urgent call about their child’s high fever, requiring an emergency room visit—should they prioritize being a dedicated researcher or a caring parent? This is a classic example of role conflict , where the expectations of multiple social roles clash and cannot all be fulfilled simultaneously. While this particular scenario might seem to have an obvious resolution favoring the child, the complexity deepens when we consider variations: what if, instead of a paper deadline, the researcher faces an imminent dissertation defense that determines their career trajectory? Such specific changes in situational urgency can dramatically shift the moral calculus, revealing how role conflicts are influenced by competing social factors, including personal preferences, cultural expectations, and situational urgency. As large language models (LLMs) are increasingly deployed in personalized advisory systems ( Takayanagi et al. 2025 ) , automated decision-making in healthcare and governance ( Lin et al. 2025 ; Kim et al. 2025a ) , and social simulation ( Vezhnevets et al. 2023 ) , they inevitably encounter role conflicts and must navigate competing social expectations. When a user asks an LLM whether to attend a crucial work meeting or take their sick friend to the hospital, the model must implicitly weigh the priorities of a responsible employee against those of a loyal friend. Understanding how LLMs resolve these competing demands—and what biases may be embedded in their decisions—is crucial for ensuring equitable AI assistance in real-world applications. However, current evaluation frameworks fall short of capturing this complexity. While previous research has examined the social abilities of LLMs, including social norms compliance ( Sap et al. 2019 ; Hendrycks et al. 2021 ; Yuan et al. 2024 ; Lee et al. 2024 ) , understanding social relationships ( Jurgens et al. 2023 ; Zhan et al. 2023 ) , and moral reasoning ( Jin et al. 2022 ; Ji et al. 2025 ; Kim et al. 2025b ) , these studies typically focus on prescriptive contexts with predetermined “correct” answers. Role conflicts, by contrast, represent inherently difficult social dilemmas without singular correct solutions, where the appropriateness of decisions depends on the contextual factors and competing preferences or value systems. This gap limits our understanding of how LLMs navigate the ambiguous, multi-faceted social scenarios they will increasingly encounter in real-world deployments. Critically, effective navigation of role conflicts requires contextual sensitivity—the ability to recognize and appropriately weigh subtle situational cues that can dramatically alter the moral calculus of a decision. For instance, returning to our researcher example, an LLM should recognize that the urgency of a dissertation defense (career-defining) versus a paper deadline (routine) fundamentally changes the decision landscape, even when the core conflict structure remains identical. We introduce RoleConflictBench , a story-based benchmark designed to evaluate an LLM’s contextual sensitivity to complex social dilemmas. To generate realistic scenarios, the benchmark employs a three-stage pipeline: (1) Expectation Generation: we first curate common social expectations for various roles; (2) Situation Instantiation: we then generate specific situations for each expectation, systematically varying their level of urgency; (3) Story Synthesis: finally, we synthesize these elements into first-person vignettes that place two roles in direct conflict. By covering nine pairs of urgency levels across the two roles, our benchmark captures a wide range of realistic, everyday role conflicts, allowing for a controlled evaluation of how LLMs weigh competing social expectations in different situations. In our experiments, we evaluate the contextual sensitivity of LLMs on RoleConflictBench , a dataset of over 13,914 role conflict scenarios centered on 65 distinct roles. We find that while current LLMs exhibit some capacity to respond to well-designed contextual cues—expectation and urgency—this sensitivity is insufficient. More significantly, their decisions are predominantly governed by powerful inherent biases related to social roles and attributes, rather than by the situational information provided. Our analysis quantifies these preferences by calculating priority scores for all 65 roles, revealing the specific roles, social attribute groups, and domains that models favor. Across most evaluated models, we observe a dominant preference for roles within the Family and Occupation domains, alongside a clear prioritization of male roles and Abrahamic religious roles. 2 Related Work Assessing Social Abilities of LLMs As LLMs are increasingly applied in diverse social contexts, research on assessing their social abilities has grown substantially. Some studies ( Forbes et al. 2020 ; Hendrycks et al. 2021 ; Yuan et al. 2024 ) have focused on social compliance, assessing how well LLMs follow social norms, while others have examined moral decision-making and reasoning, analyzing whether LLMs can make appropriate choices in various ethical or moral situations ( Emelin et al. 2020 ; Lourie et al. 2021 ; Jiang et al. 2021 ; Jin et al. 2022 ; Pyatkin et al. 2022 ; Ji et al. 2025 ; Kim et al. 2025b ) . Additional research has examined social relationships to assess whether LLMs can engage in appropriate actions or conversations in the given social context ( Jurgens et al. 2023 ; Zhan et al. 2023 ) . Other studies have assessed social commonsense ( Sap et al. 2019 ; Lee et al. 2024 ) , testing LLMs’ reasoning abilities based on everyday knowledge. A few studies have proposed evaluation frameworks, aiming to evaluate how LLMs understand social and cultural contexts and perform moral reasoning ( Zhou et al. 2023 ; Qiu et al. 2024 ) . However, prior work, which utilizes ground-truth behaviors and is limited to prescriptive contexts with “correct” answers, does not fully

Read the complete text