Randomized Controlled Trials for Security Copilot for IT Administrators

James Bono, Alec Xu · 2024

Read Randomized Controlled Trials for Security Copilot for IT Administrators by James Bono, Alec Xu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01067.txt.

Description

Abstract As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft’s Security Copilot (“Copilot”) on information technology administrators (“IT admins”) through randomized controlled trials.

Catalog record

CREATORJames Bono, Alec Xu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Randomized Controlled Trials for Security Copilot for IT Administrators James Bono, Alec Xu Abstract As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft’s Security Copilot (“Copilot”) on information technology administrators (“IT admins”) through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft’s Entra and Intune admin centers. Across three IT admin scenarios – sign-in troubleshooting, device policy management, and device troubleshooting – Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making. Keywords: Generative AI, productivity, IT admin, Security Copilot, randomized controlled trial, experiment 1 Introduction As generative AI (GAI) tools continue to mature, organizations are interested in understanding their impact on workforce productivity to guide adoption decisions. However, the effects of these tools can vary significantly depending on the specific domain and application. So, it is important to study each application individually. In this study, we evaluate the benefits of generative AI on information technology administrators (‘‘IT admins’’) by conducting a randomized controlled trial (RCT) with IT admins using Microsoft’s Security Copilot (‘‘Copilot’’). We find that Copilot confers substantial benefits to IT admins, including accuracy improvements across all scenarios and tasks (34.53%*** on average) and time savings across all scenarios (29.79%*** on average). 1 1 1 Throughout, we indicate statistical significance using asterisks, where * indicates 0.05<p\leq 0.1 , ** indicates 0.01<p\leq 0.05 , and *** indicates 0<p\leq 0.01 . As an add-on feature of Microsoft’s identity management (Entra) and device management (Intune) solutions, Copilot’s core functionality consists of its ability to reason across the vast data these solutions contain. In identity management, this includes all the existing identities, sign-in logs, conditional access policies, alerts about risky behavior, etc. In device management, this includes all the existing devices, policies, compliance states, etc. When configuring policies and troubleshooting devices and sign-ins, Copilot can extract and summarize salient information from multiple sources, giving IT admins the complete context without needing to compile the information themselves. We recruited 182 subjects and asked them to sign into an environment created for this experiment. Half of the subjects were assigned to the control group and were granted access to the Entra and Intune admin centers. The other half were assigned to the treatment group and were additionally given access to Copilot within the Entra and Intune admin centers. Then, subjects used these tools and the simulated data in that environment to respond to our experimental tasks. Our experimental tasks correspond to three scenarios in the identity management and device management domains. In the identity management domain, we study sign-in troubleshooting with tasks that mimic the kinds of activities that IT admins regularly engage in when diagnosing sign-in issues, including information retrieval and logs understanding. In the device management domain, we study both policy management and device troubleshooting. The policy management tasks focus on understanding the effects of policy settings on users and security. The device troubleshooting tasks require subjects to diagnose device non-compliance and which policies could be responsible for user-reported device issues. Our study includes multiple choice (MC), select-all-that-apply (SATA), and free response tasks. For each task, we measure completion time and accuracy. We also measure overall user satisfaction. The following is a high-level summary of our primary research questions and answers: • RQ1: Does Copilot offer productivity improvements in common IT admin scenarios? We find statistically significant improvements in speed and accuracy for all three scenarios. • RQ2: Do Copilot’s productivity effects differ by subjects’ experience levels in device management and identity management? We estimate greater effects for less experienced subjects. However, the differences are not statistically significant. • RQ3: How do Copilot’s productivity effects differ according to task type, i.e., free response versus MC and SATA? The benefits in speed and accuracy increase in the complexity of the task, with free response tasks garnering the greatest improvements. MC tasks were the least affected. • RQ4: Are subjects satisfied with Copilot’s capabilities and user experience? Copilot subjects found the task less draining and requiring less effort than the control group, and they correctly perceived speed and accuracy improvements. They also indicated they would like to use Copilot the next time they do such tasks. This paper is organized as follows. First, we review related work. Next, we provide a detailed description of our experimental methods. Then, we present the results of our analysis and address the research questions above. Finally, we discuss the implications of these findings and future work. 2 Related Work The question of GAI’s impact on productivity is central in driving both firm-level adoption decisions and economic policy decisions. Studies that examine GAI’s productivity effects from a macroeconomic perspective often arrive at different conclusions about their likely magnitude. Looking at global patent and publication data, Parteka and Kordalska (2023) fail to find a strong relationship between GAI and firm productivity. Projecting into the future, Acemoglu (2024) similarly suggests the effects will be modest, no more than a 0.66% increase in total factor productivity over 10 years. Other studies suggest the opposite. Looking at survey data on GAI adoption for a sample of German firms and controlling for endogeneity, Czarnitzki et al. (2023) find robust positive and significant associations between GAI adoption and firm productivity. Gao et al. (2023) look at micro-level manufacturing data and estimate that every 1% of GAI penetration leads to 14% increase in total factor productivity. On uptake, Bick et al. (2024) conduct a survey of US workers and find that GAI adoption has been faster than the personal computer and the internet with one in nine workers using it every workday. To complement the macroeconomic perspective, many studies have turned to measuring domain and task-specific productivity effects through laboratory or field experiments. Our work adds to this growing literature exploring GAI’s potential to improve performance across various domains and task types. The overarching theme of this literature is to provide insights into the specific conditions under which GAI delivers the most value. Like our study, most of this work shows that GAI offers significant productivity benefits to skilled workers. In the software development domain, Peng et al. (2023) conducted a controlled experiment that asked developers to complete programming tasks with and without the assistance of GitHub Copilot. The results showed a 55.8% reduction in task completion time for the group using Copilot. Similarly, Noy and Zhang (2023) observed substantial productivity improvements in professional writing tasks through the use of ChatGPT. In their experiment, GAI reduced task

Read the complete text