Randomized Controlled Trials for Conditional Access Optimization Agent
James Bono, Beibei Cheng, Joaquin Lozano · 2025
Read Randomized Controlled Trials for Conditional Access Optimization Agent by James Bono, Beibei Cheng, Joaquin Lozano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13865.txt.
Introduction
James Bono, Beibei Cheng, and Joaquin Lozano ran a controlled trial of an AI agent that helps manage access rules in Microsoft Entra. The agent supports four tasks: merging policies, finding gaps against a zero-trust baseline, planning staged rollouts, and matching users to policies. In the trial, 162 identity administrators worked with or without the agent.
Those with agent access were much more accurate overall, and they finished tasks faster at the same accuracy level. The biggest gains came on the hardest thinking tasks, such as spotting baseline gaps. The authors present the trial as early hard evidence that purpose-built agents can raise both speed and quality in identity administration.
Catalog record
| CREATOR | James Bono, Beibei Cheng, Joaquin Lozano |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13865
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Randomized Controlled Trials for Conditional Access Optimization Agent James Bono, Beibei Cheng, Joaquin Lozano Abstract AI agents are increasingly deployed to automate complex enterprise workflows, yet evidence of their effectiveness in identity governance is limited. We report results from the first randomized controlled trial (RCT) evaluating an AI agent for Conditional Access (CA) policy management in Microsoft Entra. The agent assists with four high-value tasks: policy merging, Zero-Trust baseline gap detection, phased rollout planning, and user-policy alignment. In a production-grade environment, 162 identity administrators were randomly assigned to a control group (no agent) or treatment group (agent-assisted) and asked to perform these tasks. Agent access produced substantial gains: accuracy improved by 48% and task completion time decreased by 43% while holding accuracy constant. The largest benefits emerged on cognitively demanding tasks such as baseline gap detection. These findings demonstrate that purpose-built AI agents can significantly enhance both speed and accuracy in identity administration. Keywords: Generative AI, agents, productivity, identity, conditional access, Security Copilot, randomized controlled trial, experiment 1 Introduction AI agents are increasingly deployed not just as tools but as collaborators in economic production. The administrative backbone of the digital economy – IT infrastructure – offers a fertile domain to evaluate AI’s impact. Identity management in particular sits at the junction of security and productivity: its effectiveness determines both organizational safety and the efficiency of digital workflows. The question of whether purpose-built AI agents can meaningfully augment this kind of administrative work is therefore central not only to IT operations but also to the broader debate about how AI reshapes labor and productivity. The ability to enforce finely grained conditional access (CA) policies is critical to identity security. Yet crafting, tuning, and auditing these policies is cognitively and operationally expensive. In this paper, we present the first randomized controlled trial (RCT) to evaluate whether an AI agent that suggests or optimizes CA rules – the Entra Conditional Access Optimization Agent (“the agent”) – can materially boost the productivity of identity administrators (“admins”). We randomize a diverse set of admins into Treatment and Control groups, granting the former access to the agent’s suggestions while the latter perform tasks unaided. Our results show robust gains in both task speed and correctness: administrators assisted by the agent complete tasks 43% faster while holding accuracy constant and are 48% more accurate. These findings open a path toward AI-augmented identity governance, highlighting heterogeneity of gains across task types. The agent is designed to perform a set of four complex and labor-intensive tasks associated with CA policy management. These include i) merging existing policies to reduce policy overlap; ii) identifying gaps in Zero-Trust baseline policies and building policies to fill them; iii) designing phased rollout of new policies to ensure smooth deployment and minimize operational impacts; and iv) ensuring newly added users are included in appropriate existing policies to achieve Zero-Trust coverage. The agent can be configured to make suggestions only or to autonomously create new policies in report-only mode for an admin to review and approve. We recruit a targeted population of 162 identity admins to perform the four CA policy management tasks in a purpose-built, production-grade environment. Subjects interact with the environment and record their responses to the tasks via an instrumented survey platform. Subjects are split into two groups: a Control group without access to the agent and a Treatment group that has access to the agent’s suggestions. Research Questions. Our research questions are motivated by what a CIO would want to know prior to adoption: • RQ1 (Accuracy): Does using the agent improve the quality of admin decisions? • RQ2 (Speed): Holding accuracy constant, how much time does the agent save an admin? AI’s measured productivity effects are now well-documented across writing, customer support, and software development, with sizable average gains but marked heterogeneity by task and worker experience ( 14 , 7 , 15 , 10 ) . Within enterprise IT operations, evidence likewise points to faster, more accurate work from generative assistants ( 6 , 11 ) and associated improvements in live operational metrics ( 5 , 4 ) . Moving from copilots to agents, some expect planning, memory, and tool use expand the scope for automation and complementarity ( 13 ) . Agent experiments, like the present one, have begun to quantify agent effects in collaborative and technical workflows, highlighting design, trust, and integration frictions ( 12 , 9 , 2 ) . However, from a macroeconomic perspective, rapid adoption coexists with tempered forecasts for aggregate total factor productivity gains absent complementary organizational change ( 3 , 1 , 8 ) . Against this backdrop, our contribution is a field-grade RCT in identity governance that measures how a purpose-built agent shifts speed and accuracy for professional administrators. This paper is organized as follows. First, we provide a detailed description of our experimental methods. Then, we present the results of our analysis and address the research questions above. Finally, we discuss the implications of these findings and future work. 2 Methodology We follow a standard RCT protocol to identify and estimate the causal effect of the agent on accuracy and task completion time. That is, we randomly assign half of our subjects to the Control group, which only allows them use of the Entra admin center without the agent. The other half are assigned to the Treatment group, which gives them the additional ability to use the agent’s output when responding to the experimental tasks. To make our design as realistic as possible, we placed no restrictions on how subjects used other tools, such as other AI tools or web searches. Therefore, our measurements reflect the incremental benefit of the agent on top of the admin centers and other tools like web searches and other AI. We recruited our subject pool through Upwork, a marketplace for freelancers. We required subjects to have some experience in identity management, to be proficient in reading and writing English, and to have a positive reputation on Upwork. We told them to expect the tasks to take up to 1.5 hours to complete. We offered performance incentives for combined speed and accuracy. Specifically, we multiplied the percentiles of each subject’s speed and accuracy and awarded payments on the following scale: • Top 10% in combined speed and accuracy: $85 • Top 10-20% in combined speed and accuracy: $70 • Top 20-30% in combined speed and accuracy: $55 • Completed in good faith but did not reach the top 30% in combined speed and accuracy: $35 • Show up fee: $15 We had 162 subjects complete the task. Our subject pool embodied a high level of experience across identity management. Table 1 presents a breakdown of subjects’ self-reported experience level by experimental group. We note that the groups are not perfectly balanced by experience level. However, when including experience level in regressions to measure speed and accuracy differences, we only find a statistically significant effect for the accuracy metric for one of our tasks – Policy Merge. Therefore, our results on Policy Merge accuracy control for experience level. We do this for completeness only and note that the results are robust to this choice. Table 1 : Experience Distribution Across Control and Treatment Groups Experience Level