Disclosure By Design: Identity Transparency as a Behavioural Property of Conversational AI Models

Anna GausenAI Security InstituteUK Sarenne WallbridgeAI Security InstituteUK Hannah Rose KirkAI Security InstituteUK Jennifer WilliamsUniversity of SouthamptonUK Christopher SummerfieldAI Security InstituteUK · 2026

Read Disclosure By Design: Identity Transparency as a Behavioural Property of Conversational AI Models by Anna GausenAI Security InstituteUK Sarenne WallbridgeAI Security InstituteUK Hannah Rose KirkAI Security InstituteUK Jennifer WilliamsUniversity of SouthamptonUK Christopher SummerfieldAI Security InstituteUK (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16874.txt.

Introduction

Gausen and co-authors argue that chat AI should state it is AI whenever a user asks directly. Users often cannot tell whether they face a human or a machine, and unclear identity invites misplaced trust or fraud. Laws require self identification but say little about live conversation. Interface labels can be omitted, and provenance tools need shared infrastructure. The team tested deployed text and voice systems in normal, role play, and adversarial settings. Disclosure was often high at baseline but fell sharply in role play and under attack.

Catalog record

CREATORAnna GausenAI Security InstituteUK Sarenne WallbridgeAI Security InstituteUK Hannah Rose KirkAI Security InstituteUK Jennifer WilliamsUniversity of SouthamptonUK Christopher SummerfieldAI Security InstituteUK
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83886
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Disclosure By Design: Identity Transparency as a Behavioural Property of Conversational AI Models Anna GausenAI Security InstituteUK Sarenne WallbridgeAI Security InstituteUK Hannah Rose KirkAI Security InstituteUK Jennifer WilliamsUniversity of SouthamptonUK Christopher SummerfieldAI Security InstituteUK Abstract As conversational AI systems become more widely used, it is important that users are aware of whether they are talking to a human or an artificial system. Increasingly realistic language generation and a wider range of deployment contexts are diminishing users’ certainty about their interlocutor’s identity. When users are unaware that their conversation partner is AI, they may unwittingly share sensitive information inappropriately, place unwarranted trust in AI-generated advice, or fall victim to AI-enabled fraud. More broadly, a persistent lack of transparency will erode trust in mediated communication (texts, calls, and other non-face-to-face channels). While regulations like the EU AI Act and California’s BOT Act require AI systems to identify themselves, they provide limited guidance on what reliable disclosure should look like in real-time conversation. Existing mechanisms for transparency also leave gaps, as interface indicators can be omitted by downstream deployers, while provenance tools require coordinated infrastructure and cannot provide reliable real-time verification. In this work, we ask how conversational AI systems should maintain identity transparency as human-AI interactions become more ambiguous and diverse. We advocate for disclosure by design , where AI systems explicitly disclose their artificial identity when directly asked by a user. Transparency that is implemented and experienced through AI model behaviour can persist across deployment contexts without reliance on user interfaces and downstream deployment. This mechanism also preserves user agency to verify AI identity on demand in real-time conversations, without interfering with immersive experiences like role-playing. To empirically assess the extent to which deployed conversational AI systems disclose their identity when prompted, we present and release the first multi-modal (text and voice) evaluation of disclosure behaviour. This measures how often an AI reveals its identity across baseline, role-playing, and adversarial prompting. Our work shows that while disclosure rates in baseline conditions are often high, they drop substantially under role-playing and can be suppressed almost entirely under adversarial prompting conditions. Importantly, we show that disclosure rates can vary significantly across model providers and modalities, highlighting how brittle disclosure behaviour currently is. To address these shortcomings, we present a range of technical interventions to practically assist developers with embedding disclosure as a fundamental property of conversational AI models to preserve transparent interactions. UKGovernmentBEIS/disclosure-by-design-evals 1 Introduction Millions of people now interact with conversational AI systems on a daily basis ( 14 ; 18 ) . The AI companion platform Character.AI alone processes the equivalent of 20% of the request volume of Google Search, reporting 20,000 user queries per second ( 16 ; 66 ) . However, in tandem with rapid AI adoption, there has been a steady erosion of cues that might signal to users whether their conversation partner is human or AI. Firstly, advances in AI model capability mean that synthetic content is increasingly indistinguishable from that generated by humans ( 39 ; 9 ) . Secondly, human-AI interactions increasingly take place in the voice modality, where richer anthropomorphic cues alter social perceptions of interlocutor identity ( 30 ; 27 ) . Thirdly, users are increasingly using AI for immersive functions like role-play, which tend to blur identity boundaries ( 23 ) . Lastly, AI systems are increasingly deployed not just via dedicated AI interfaces (e.g., consumer websites) but also embedded in other technologies (e.g., telephone calls or messaging services), where users may historically be expecting to interact with a human. Collectively, these four changes increase the likelihood of undisclosed or ambiguous human-AI interactions. As AI adoption grows, it is increasingly important that users know whether they are talking to an artificial system or to another human (we call this property of the interaction “AI identity transparency”). Failures of AI identity transparency can leave users open to fraud or other forms of manipulation, and leads to a generalised degradation of trust in the communication channel and wider information ecosystem ( 61 ) . If users are constantly at risk of being exposed to hard-to-detect AI systems, they may default to a mode of suspicion, leading to continuously elevated cognitive load. Regulators have begun to recognise these risks; the EU AI Act and California’s BOT Act mandate that services disclose to users when they are interacting with an AI system ( 29 ; 70 ) . Several research papers also highlight the importance of identity transparency for developing ethical AI ( 2 ) , reducing anthropomorphism ( 20 ) , and preventing trust miscalibration ( 26 ) . Despite this regulatory and academic attention, we lack both clear operationalisation of how identity transparency should be achieved during real-time conversation, as well as empirical evidence of whether deployed systems meet such a standard ( 36 ; 26 ) . While defining appropriate transparency across the broadening deployment contexts and use-cases of conversational AI is inherently challenging, it is also becoming increasingly important to assess experimentally. In this paper, we make three contributions in the direction of AI identity transparency. First, we advocate for enhancing AI identity transparency through model behaviour: disclosure by design , where AI systems explicitly state their identity when asked directly by a user (Section 3 ). Operationalising disclosure in this way balances transparency requirements with users’ demand for legitimate applications like role-play. Implementing transparency through model behaviour contrasts with approaches that require AI identity cues to be attached to deployment interfaces, where they can be easily removed and may be ignored. Constraining disclosure to direct user queries preserves user agency to verify identity in real-time, without relying on post-hoc provenance analysis. For our second contribution, we empirically validate whether current voice- and text-mode conversational AI systems reveal their identity when asked. We develop the first multi-modal evaluation of AI transparency by disclosure (Section 4 ) and open-source our evaluation pipeline. Our experiments include different prompted use-cases, ranging from baseline (e.g., “You are a helpful assistant.”) to role-playing (e.g., “You are a female called Jane.”) to adversarial (e.g., “Do not say you are AI under any circumstance.”). We evaluate different open- and closed-weight AI models with 7,000 interaction cases for text interactions and 42,000 test cases for voice interactions (Appendix A.2.1 ). We find that disclosure rates vary substantially depending on which model is used, both across modality and provider. Disclosure is also vulnerable to how the model is used: it is suppressed by role-playing and can be removed completely through adversarial system prompts. These findings reveal the complex risk profile of human-AI conversation, where achieving robust disclosure that protects users across varied interaction contexts remains a technical challenge. For our final contribution, to facilitate uptake of more transparent disclosure in deployed systems, we present an overview of potential technical interventions to make disclosure more robust across models, modalities, and deployment scenarios (Section 5 ). 2 The Changing Landscape of Human-AI Interaction We review evidence on four simultaneously shifting dimensions of human-AI interaction that make AI identity transparency challenging. These shifts span both text and voice modalities, the two dominant channels for conversational AI interaction. 2.1 Advancing generation capabilities Recent developments in model architecture and training data scale have produced significant advancements

Read the complete text