Concept Incongruence: An Exploration of Time and Death in Role Playing
Xiaoyan Bai, Ike Peng *, Aditya Singh *, Chenhao Tan, University of Chicago · 2025
Read Concept Incongruence: An Exploration of Time and Death in Role Playing by Xiaoyan Bai, Ike Peng *, Aditya Singh *, Chenhao Tan, University of Chicago (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14905.txt.
Introduction
Four authors, Xiaoyan Bai, Ike Peng, Aditya Singh, and Chenhao Tan, name a flaw they call concept incongruence. Their 2025 paper starts with a simple clash: a user asks for a unicorn with two horns, though a unicorn has one horn by its very meaning. The team tests the flaw in role play, where a model acts as a person from the past.
The authors score models on whether they stop, stay right, and answer at all once the played person has died. Models kept going after the death point and lost ground on facts. Tests inside the models point to weak coding of the death state. Fixes based on those tests made the models act in a steadier way.
Catalog record
| CREATOR | Xiaoyan Bai, Ike Peng *, Aditya Singh *, Chenhao Tan, University of Chicago |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concept Incongruence: An Exploration of Time and Death in Role Playing Xiaoyan Bai, Ike Peng *, Aditya Singh *, Chenhao Tan, University of Chicago Abstract Consider this prompt “Draw a unicorn with two horns”. Should large language models (LLMs) recognize that a unicorn has only one horn by definition and ask users for clarifications, or proceed to generate something anyway? We introduce concept incongruence to capture such phenomena where concept boundaries clash with each other, either in user prompts or in model representations, often leading to under-specified or mis-specified behaviors. In this work, we take the first step towards defining and analyzing model behavior under concept incongruence. Focusing on temporal boundaries in the Role-Play setting, we propose three behavioral metrics—abstention rate, conditional accuracy, and answer rate—to quantify model behavior under incongruence due to the role’s death. We show that models fail to abstain after death and suffer from an accuracy drop compared to the Non-Role-Play setting. Through probing experiments, we identify two main causes: (i) unreliable encoding of the ‘‘death’’ state across different years, leading to unsatisfactory abstention behavior, and (ii) role playing causes shifts in the model’s temporal representations, resulting in accuracy drops. We leverage these insights to improve consistency in the model’s abstention and answer behaviors. Our findings suggest that concept incongruence leads to unexpected model behaviors and point to future directions on improving model behavior under concept incongruence. 1 1 1 Our code is available at: https://github.com/ChicagoHAI/concept-incongruence.git . * * footnotetext: These authors contributed equally to this work. 1 Introduction Large language models provide a simple interface for anyone to control their behavior through arbitrary natural language instructions [ 5 , 26 , 36 ] . Such a general interface often leads to “conflicting” demands. An example is asking the model to role-play Marilyn Monroe (d. 1962) while simultaneously requesting information about current politics. The lifespan of the character clashes with the expectation that the agent knows political information from the twentieth century. We call such clashes concept incongruence : two or more concept boundaries specified (implicitly or explicitly) in the prompt are incongruent with each other. We propose three levels of concept incongruence (illustrated in Figure 1 ): I-A Between human concepts in the prompt. In addition to the unicorn example, another one can be “Propose a system where prices are determined by market forces but always remain stable.” , a seemingly reasonable task yet impossible to complete. This is an instance of mis-specification because it is impossible to complete the task without resolving the incongruence, although ChatGPT generates “a unicorn with two horns” anyway. I-B Between human concepts in the prompt and the model’s internal representations. An example is “Say green when seeing purple”, a well-studied challenging task for humans known as the Stroop effect [ 34 ] . 2 2 2 The Stroop effect refers to the delay in reaction time between neutral and incongruent stimuli, and is often used to investigate a person’s psychological capabilities [ 20 ] . In this case, this prompt can be correctly followed, but the model’s internal representation of green and purple may clash and cause undesirable behavior. The above Marilyn Monroe example also belongs to this category, and can benefit from clarifying what knowledge the user would like Marilyn Monroe to be equipped with. I-C Between internal representations that the model activates. For example, a model’s internal representations of harmless and helpful may clash when asked to provide instructions on bypassing website restrictions even though these two concepts are not explicit in the prompt, leading to under-specified behavior. Such incongruence often occurs in alignment faking [ 9 ] and jailbreaking attempts [ 40 ] , and represents genuine challenges in AI safety, as desirable behavior is often context-dependent and user-specific. Figure 1 : An illustration of three levels of incongruence. (A): Impossible to complete without resolving the clash ( mis-specification ), although ChatGPT proceeds to generate an image; (B): Possible to complete but challenging for the models. It is relatively easy to trace the incongruence because incongruence shows up in the prompt. It could benefit from specification, as in the Marilyn Monroe example ( under-specification ); (C): Challenging to trace the incongruence because the incongruence does not show up in the prompt. It is also hard to specify the desirable behavior ( under-specification ). In this study, we examine concept incongruence in the Role-Play setting, an instantiation of I-B . Adopting a role grants the model a concept boundary, which is defined as the set of facts confined by that role’s timeline, in particular, the role should not know events after their death. Open-ended user prompts (e.g., asking questions after a role’s death) push the model beyond this boundary, creating the kind of incongruence defined above. This setting affords an appropriate interpretation for completing the instruction and allows us to investigate two key questions: (1) How do models behave when concept incongruence occurs? (2) How do these behaviors emerge from model representations? To do that, we introduce three metrics for quantifying model behavior: abstention rate, answer rate, and conditional accuracy. We find that current LLMs rarely abstain from questions after death; even when they try to do so, the abstention rate does not change sharply around death time. Moreover, there is an intriguing accuracy drop in the Role-Play mode compared to Non-Role-Play . Next, we locate the source of the incongruence with the probing experiments. We show that the model lacks a reliable representation of the death state, especially the death year, resulting in the above behavior. These gaps make the internal representation of the model incongruent with the human’s representation of Role-Play ( I-B ). Additionally, we demonstrate evidence that the Role-Play mode causes shifts in the model’s temporal representations, leading to inconsistent world knowledge. This finding suggests implicit clashes within model representations between role playing and world knowledge, an instantiation of I-C that we did not expect. We further improve model behavior with these insights by providing additional specification and conclude with a discussion of future directions for studying and managing model behavior under concept incongruence. In summary, we make the following contributions: • We introduce concept incongruence and provide the first systematic categorization to illustrate the space of problems. • We create a benchmark centered on time and death in role playing and show that current models do not demonstrate desirable abstention behavior and present a drop in accuracy when role playing. • We find that the inconsistent behavior emerges due to the lack of reliable representations of death and the clash between role playing and world knowledge in the model’s internal representations. 2 Experiment Setup In this section, we first explain the key evaluation metrics of interest and then introduce implementation details, including our dataset and prompt design. 2.1 Behavioral Metrics Our key behavior of interest is how an LLM handles questions after death when it is playing a character. The concept boundary, due to the character’s life, and the concept boundary in the time of the question lead to concept incongruence. Next, we introduce our evaluation metrics to quantify the behavior of interest and our expected behavior. Abstention Rate. Since each character has a knowledge boundary, for questions that fall outside this