Pre‐implementation safety evaluation of an AI decision‐support system for surgical antimicrobial prophylaxis Authors: Joshua Andrews, Sanjat Kanjilal, Camile Kotton, Brandon Stretton, Alasdair Leslie Published: 2026 ABSTRACT AbstractBackgroundLarge language models (LLMs) show potential to support antimicrobial prescribing but require simulation‐based, institution‐specific safety evaluation prior to any consideration of clinical use. In Australia, antimicrobial prescribing represents a high‐risk domain for digital decision‐support systems due to patient safety and antimicrobial resistance implications.AimTo characterise prescribing accuracy, error phenotypes and antimicrobial stewardship risk associated with a LLM that was provided with publicly available surgical prophylaxis guidelines during inference (without fine‐tuning or model modification) across 20 simulated surgical scenarios.MethodsTwenty simulated surgical scenarios were tested using a LLM that was prompt‐conditioned with publicly available guideline text during inference, without any fine‐tuning or modification of model weights. For each case, the model generated recommendations for agent, dose, timing, re‐dosing and guideline citation. Outputs were independently assessed by two local clinicians familiar with the guideline, with accuracy scored across five domains and harm classified using a modified National Coordinating Council for Medical Error Reporting and Prevention (NCC MERP) Index.ResultsClinically significant antimicrobial prescribing risk was identified in 10% of simulated scenarios (2/20), recognising wide confidence intervals due to the small sample size. These included omission of required anaerobic coverage and failure to redose prophylaxis in prolonged procedures. Overall guideline concordance was 4/5, with perfect dose accuracy but lower performance for timing (70%) and guideline citation (45%).ConclusionsThis study demonstrates the feasibility of constructing institutionally governed, guideline‐based AI systems while identifying stewardship‐relevant safety risks that currently preclude clinical use without further validation. Introduction Artificial intelligence (AI), particularly large language models (LLMs), has rapidly evolved from experimental tools into emerging clinical decision‐support systems. These models can synthesise data, streamline documentation and generate structured recommendations. Earlier models such as GPT‐4 showed strong reasoning capability across a range of medical tasks, including diagnostic support, clinical communication, documentation, antimicrobial prescribing and medical knowledge benchmark exams. 1 , 2 , 3 , 4 , 5 , 6 Recent studies suggest that newer LLMs achieve higher accuracy than prior models in multimodal medical reasoning 7 and can generate reliable diagnostic support for chronic disease management, including diabetes. 8 While these developments highlight the growing use of LLMs in clinical contexts, safe, institution‐specific implementation remains essential. The transition from potential to structured use has already begun, as evidenced by the successful embedding of GPT‐4 into medical education curricula. 9 The next logical step is a careful, evidence‐based assessment of safety and governance requirements prior to any consideration of clinical deployment. Even infrequent but plausible antimicrobial prescribing errors generated by such systems could propagate rapidly and at scale, with disproportionate consequences for patient safety and antimicrobial resistance, particularly if deployed without formal governance or validation. From a health‐system perspective, such risks are particularly relevant in Australia, where antimicrobial stewardship programmes operate within nationally coordinated governance frameworks. Pre‐implementation evaluation of digital prescribing tools is therefore essential to align innovation with patient safety and stewardship obligations. Antimicrobial prophylaxis provides an ideal testing environment for this development as it is a structured, guideline‐driven task requiring precision in agent selection, dose, timing and re‐dosing, all of which are critical to patient safety. 10 Despite established recommendations, adherence to prophylaxis timing and agent selection remains inconsistent, with international data suggesting suboptimal compliance rates in tertiary surgical settings. 11 , 12 These deviations often arise from difficulty accessing or interpreting local guidelines during time‐critical perioperative workflows. 13 Establishing local validation and accountability is therefore a critical next step toward safe, contextualised clinical deployment. A locally fine‐tuned LLM capable of generating instant, guideline‐specific recommendations could theoretically reduce cognitive load and support antimicrobial stewardship, but only if safety, reliability and governance requirements are met. As clinicians increasingly access generative AI tools informally, defining safety boundaries and stewardship‐relevant errors is essential to prevent ungoverned adoption prior to clinical validation. This study aimed to conduct a pre‐implementation safety and governance‐focused evaluation of a guideline‐based AI decision‐support tool for antimicrobial prophylaxis, focusing on accuracy, errors and potential prescribing harm in simulated clinical scenarios. The model was developed entirely from publicly available guidelines, without external datasets or fine‐tuning, demonstrating that locally relevant LLMs can be rapidly configured using existing resources. This evaluation was designed as a stewardship‐focused safety and governance risk assessment rather than a validation of clinical effectiveness or readiness. To our knowledge, no Australian study has systematically evaluated antimicrobial stewardship risk associated with LLM decision support prior to clinical deployment. Methods Study design This was a single‐arm descriptive evaluation of one LLM using 20 simulated surgical prophylaxis scenarios. The outcomes were accuracy, clarity and harm classification. Two local clinicians with experience using the guidelines independently assessed all outputs, with consensus resolution for discrepancies. Simulation‐based testing is a standard approach in medication safety research, allowing identification of error patterns and failure modes before clinical exposure. Given the ethical and governance implications of deploying AI decision‐support tools without prior safety evaluation, a simulation framework was selected to characterise risks under controlled conditions. This approach mirrors pre‐clinical safety testing commonly used in medication error and stewardship research prior to workflow integration. The evaluation was designed to inform governance and stewardship decision‐making rather than to assess clinical effectiveness. LLM model, version and access method A contemporary LLM (GPT‐5; OpenAI, San Francisco, CA, USA) was accessed via the publicly available ChatGPT web interface in August 2025. No Application Programming Interface (API) access, fine‐tuning, retrieval augmentation or modification of model weights was performed. All outputs reflect the behaviour of the base model using only the supplied contextual documents and prompts. Guideline ingestion and LLM configuration All publicly available surgical prophylaxis guideline PDFs were uploaded directly into the LLM prior to testing (listed in Supporting Information). The uploaded documents remained available to the model throughout the session and were used as contextual reference material during inference. No external datasets, retrieval systems or additional training was applied. Surgical antimicrobial prophylaxis guidance published through the South Australian Department for Health and Wellbeing clinical guideline repository was used as the reference standard. These publicly available documents, titled Surgical Antimicrobial Prophylaxis Prescribing Guidelines, are accessible via the SA Health website. 14 The guideline is publicly accessible without restriction and is cited directly to ensure transparency and reproducibility of the evaluation. The guideline is based on the Australian Therapeutic Guidelines, with local adaptations reflecting regional antimicrobial resistance patterns, formulary considerations and patient population characteristics. These differences typically involve modification of antimicrobial selection, dosing and prophylaxis protocols in line with local resistance patterns and formulary policies, rather than substantive deviation from national guideline principles. In Australian healthcare settings, antimicrobial stewardship programmes frequently operationalise national Therapeutic Guidelines through locally implemented protocols that incorporate formulary restrictions, local microbiology and workflow considerations. The aim of this study was to evaluate model performance against locally implemented guidelines used in routine clinical practice. Future evaluations may incorporate therapeutic guidelines as a complementary comparator. The specific guideline categories used are listed in the Supporting Information. Simulated scenarios and prompt structure Twenty simulated surgical scenarios were developed to represent common procedures across various surgical specialties. Scenarios were developed through multidisciplinary discussion between clinicians from various specialities. Procedure selection was informed by institutional prescribing frequency and representation across major surgical specialities within a tertiary referral centre. Each scenario included patient age, body mass index, renal function, allergy status, surgical procedure, surgical plan and MRSA screening status, with the primary question of recommended antimicrobial prophylaxis. Scenarios did not explicitly include rare co‐morbidities, recent overseas travel or hospitalisation, or multidrug resistant organism colonisation. These represent important real‐world modifiers and are discussed as limitations. Each case was submitted to the LLM as a separate structured prompt containing the above details. Prompts were submitted once per scenario, and responses were not regenerated. To maintain consistency, all prompts followed a template and the explicit instruction to provide antimicrobial agent, dose, timing, re‐dosing interval, time to generate and guideline citation. A representative example is shown below in Box 1. Box 1Example promptFieldContentSpecialtyOrthopaedic surgeryProcedureElective total knee replacement (expected duration 3 h)Patient 67‐year‐old female; BMI 42; CKD Stage 3a (eGFR 55 mL/min); MRSA negative; no known drug allergies; tourniquet likely Clinical questionWhat surgical antibiotic prophylaxis do you recommend according to the local guideline?Full prompt inputTom from Ortho: elective total knee replacement for a 67‐year‐old with BMI 42, CKD Stage 3a (eGFR≈55), MRSA negative, no allergies, tourniquet likely, expected duration about 3 h with possible re‐dosing need, what single regimen do you recommend for prophylaxis including weight/eGFR‐appropriate dosing, timing and re‐dosing threshold? Please state time to generate and the guideline used for the recommendation. Note: Example of a structured prompt used to query the guideline‐based artificial intelligence system for surgical antimicrobial prophylaxis. The box outlines the clinical specialty, procedure details, patient characteristics including age, body mass index, renal function, allergy status and surgical considerations, followed by a natural‐language clinical question requesting antimicrobial agent selection, dose, timing, re‐dosing guidance, generation time and guideline citation according to local institutional policy.Abbreviations: BMI, body mass index; CKD, chronic kidney disease; EGFR, estimated glomerular filtration rate; MRSA, methicillin resistant Staphylococcus aureus. Example prompt 67‐year‐old female; BMI 42; CKD Stage 3a (eGFR 55 mL/min); MRSA negative; no known drug allergies; tourniquet likely Note: Example of a structured prompt used to query the guideline‐based artificial intelligence system for surgical antimicrobial prophylaxis. The box outlines the clinical specialty, procedure details, patient characteristics including age, body mass index, renal function, allergy status and surgical considerations, followed by a natural‐language clinical question requesting antimicrobial agent selection, dose, timing, re‐dosing guidance, generation time and guideline citation according to local institutional policy. Abbreviations: BMI, body mass index; CKD, chronic kidney disease; EGFR, estimated glomerular filtration rate; MRSA, methicillin resistant Staphylococcus aureus. The instruction to provide a ‘single regimen’ was deliberately included to avoid the generation of multiple alternative options and to reflect real world perioperative workflows, where a single actionable recommendation is typically required. This constraint was applied uniformly across all scenarios to standardise output structure. Model outputs and outcome measures For each case the LLM produced a prophylaxis recommendation including antimicrobial agent, dose, timing, re‐dosing interval, guideline citation and time to generate. Outputs were evaluated against the official institutional guideline and scored in five binary accuracy domains (1 = correct, 0 = incorrect, total score 0–5). For each scenario, the expected guideline‐National Coordinating Council for Medical Error Reporting and Prevention concordant recommendation was predefined prior to model testing and is provided alongside model outputs in the Supplement to allow transparent comparison. Clarity was scored 1–5 based on readability, logical flow and ease of clinical interpretation on a subjective scale (1 = unclear, 5 = fully interpretable). Harm classification used the modified National Coordinating Council for Medical Error Reporting and Prevention (NCC MERP) index (Table 1). The NCC Medication Error Index, developed by the NCC MERP, is a standardised scoring system to categorise medication errors by the severity of the outcome. The index uses categories A–I; however, given the nature of this study, we modified the index to use A–F. Categories relating to permanent harm or death (G–I) were excluded as not applicable in simulation. For interpretive clarity, harm was stratified into non‐clinically significant harm, which included any deviation from the guideline (Categories A–D) and clinically significant prescribing risk (Categories E–F). Category D events were classified as non‐clinically significant in this simulation context because no patient exposure or clinical intervention occurred. This distinction ensures that non‐clinical or formatting‐related discrepancies are not over‐interpreted as safety risks. Table 1Adapted NCC MERP indexCategoryDescriptor (adapted for GPT output)Example N Non‐clinically significantANo error; correct recommendationFully matches publicly available institutional guideline5 (25%)BError, but unlikely to cause harmCitation mismatch or formatting issue7 (35%)CError that could cause minor harm if followedIncorrect timing (e.g., >30 min off) or unnecessary second agent6 (30%)DError that could cause temporary or reversible harmMissed re‐dosing in long procedure0Clinically SignificantEError that could cause or prolong infection / major morbidityOmission of prophylaxis or use of wrong antibiotic class2 (10%)FError that could result in severe harm or life‐threatening infectionWrong agent in penicillin‐allergic patient0 Note: Adapted National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) Index used to classify antimicrobial prescribing errors generated by the artificial intelligence system. The table categorises outputs from no error (Category A) through non‐clinically significant errors (Categories B–D) and clinically significant errors (Categories E–F), with descriptors, examples relevant to surgical antimicrobial prophylaxis and the number and percentage of simulated cases assigned to each category. Adapted NCC MERP index Note: Adapted National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) Index used to classify antimicrobial prescribing errors generated by the artificial intelligence system. The table categorises outputs from no error (Category A) through non‐clinically significant errors (Categories B–D) and clinically significant errors (Categories E–F), with descriptors, examples relevant to surgical antimicrobial prophylaxis and the number and percentage of simulated cases assigned to each category. Statistical analysis Proportions were calculated for each accuracy and harm domain. Exact 95% confidence intervals (CIs) were derived using the Clopper–Pearson method for binomial data, given the small sample size (n = 20). Descriptive statistics are reported as mean ± standard deviation or proportion (95% CI) as appropriate. No inferential tests were applied, as the study aimed to benchmark model performance descriptively rather than compare independent groups. Reproducibility considerations As the hosted LLMs such as GPT‐5 undergo model updates, identical replication of outputs may not be possible without access to locked‐version instances. This limitation was predefined and is reported in accordance with the British Medical Journal's LLM reporting recommendations. All outputs were generated using single‐pass inference without iterative prompting. Model temperature and other generation parameters were not user configurable via the web interface and therefore remained at platform defaults. Ethics This simulation‐based study involved no patient data and no clinical deployment and was therefore exempt from Human Research Ethics Committee review. Results Across 20 simulated cases, clinically significant antimicrobial prescribing errors were observed in two scenarios (10%), with the remainder containing either no error or non‐clinically significant deviations. The mean total accuracy score was 4/5 (SD 0.7). Agent selection and dose accuracy were high (90%–100%; Figure 1). Timing accuracy was lower (70%), primarily due to missed distinctions between 120‐ and 60‐min pre‐incision intervals when multiple antibiotics were required. Guideline citation accuracy was 45%, representing citation errors, which may be clinically relevant depending on context. No hallucinated antibiotic classes or dosing were identified. No dosing errors were observed across any simulated scenarios. Moderate variability was observed in timing and guideline citation domains, while re‐dosing and agent selection remained highly reliable (≥95% correct). Mean generation time was 10.2 s (SD 0.9, range 9–12). All cases scored 5/5 for clarity as assessed by two clinician reviewers. Accuracy of antimicrobial prophylaxis GPT model across five surgical prophylaxis domains. Bar chart displaying the accuracy of the antimicrobial prophylaxis artificial intelligence model across five prescribing domains: antimicrobial agent selection, dose accuracy, timing of administration, re‐dosing and guideline citation. Accuracy is shown as the percentage of correct recommendations across 20 simulated surgical scenarios, with high performance in agent selection and dosing and lower performance in timing and guideline citation. Harm analysis was conducted using the NCC MERP Index (Table 1). Across 20 cases, 75% (N = 15) contained at least one deviation, with 87% being non‐clinically significant. Clinically significant harm occurred in 10% of cases (2/20; 95% CI 1.2–31.7), while minor or non‐clinically significant errors (categories A–C) accounted for 90% (18/20; 95% CI 68.3–98.8). The two cases classified as major harm involved omission of metronidazole for anaerobic coverage during a limb amputation for critical ischaemia and complete omission of cefazolin re‐dosing following cardiac surgery. Both error types are recognised contributors to preventable surgical site infection in high‐risk procedures. Overall, most outputs were concordant with guideline recommendations, but 10% posed a plausible risk of clinically meaningful harm if enacted, indicating that apparent accuracy alone is insufficient as a safety metric. Guideline citation accuracy was substantially lower than performance in other domains. Although citation errors may appear clinically minor, they have important implications for auditability, medico‐legal defensibility and antimicrobial stewardship governance. Decision‐support systems intended for institutional use must allow clinicians to verify recommendations against authoritative sources. An inaccurate or incomplete citation therefore represents a governance‐relevant failure mode rather than a purely technical limitation. Discussion This pre‐implementation evaluation demonstrates that institutionally governed, guideline‐based AI systems can be constructed using publicly available antimicrobial prophylaxis guidance, but it also reveals clinically meaningful stewardship risks. Although overall guideline concordance was high, a clinically significant prescribing error rate of 10% was observed, highlighting that apparent accuracy alone is an insufficient safety metric for antimicrobial decision support. From an infection prevention perspective, even infrequent prescribing errors in surgical prophylaxis may contribute disproportionately to preventable surgical site infections. These findings have relevance for Australian health services currently exploring generative AI tools for clinical decision support. They highlight that apparent accuracy alone is insufficient as a safety metric and that antimicrobial stewardship‐relevant failure modes must be identified before any consideration of clinical integration. The clinically significant error rate of 10% (95% CI 1.2%–31.7%) should be interpreted cautiously. The wide confidence interval reflects the small sample size and means the true rate could be substantially higher. In high‐volume clinical environments, even a low‐frequency error rate of this magnitude would be unacceptable. Real‐world variability in patient demographics and clinical workflows may increase the likelihood of errors beyond what was observed in these controlled simulated scenarios. This evaluation was designed to identify plausible stewardship‐relevant failure modes rather than to estimate a precise error rate, consistent with pre‐implementation safety testing paradigms. Comparable studies evaluating general‐purpose LLMs for antimicrobial prescribing have reported accuracy rates between 71.7% and 96.7% across domains such as antimicrobial selection, dosing, duration and overall guideline concordance, 15 as well as accuracy in recommending correct antibiotic class, dose and duration when compared with general practitioner prescribing. 16 Descriptive comparison with prior LLM antimicrobial prescribing studies provides contextual reference; however, the small sample size and simulation design preclude meaningful quantitative benchmarking. The observed 10% clinically significant error rate should therefore be interpreted as identification of plausible stewardship relevant failure modes rather than a stable performance estimate. The model's potential as a clinical support tool is most apparent for junior or ward doctors, who frequently encounter uncertainty regarding local antibiotic protocols or experience difficulty accessing relevant guidelines in time‐critical situations. In such settings, a locally configured LLM integrated into hospital intranet or electronic medical record (EMR) systems could theoretically support access to guideline‐based recommendations, thereby reducing cognitive load, improving adherence to antimicrobial policies and supporting stewardship programmes. 6 , 16 , 17 We note that integration into hospital intranet or EMR systems requires substantial operational, privacy and governance review, which was outside the scope of this study. AI‐generated recommendations may not account for rare neuromuscular or pharmacological contraindications unless explicitly prompted. As with any decision support tool, clinician verification remains essential, particularly when users may be unaware of comorbid conditions influencing antibiotic safety. Effective implementation would also require user training regarding prompt structure, scope limitations and verification expectations. If structured prompting requires substantial manual data entry or repeated clarification, workflow efficiency may be reduced. As with other electronic antimicrobial stewardship initiatives, adoption will depend not only on accuracy but also on demonstrable time efficiency. Emerging models such as GPT‐5 further enhance multimodal reasoning and factual accuracy. 7 , 8 Coupled with evidence demonstrating successful integration of LLMs into structured educational and clinical training contexts, 9 these advances point towards a future in which supervised, governance‐aligned AI systems complement clinical decision‐making while maintaining institutional oversight. Healthcare‐oriented LLM interfaces may improve clinical alignment, but independent local validation against institutional policy remains essential, irrespective of model branding or vendor safety claims. This study provides a proof of concept for institutionally governed LLM configuration, showing that rapid, low‐cost model adaptation is feasible using existing guideline data. These findings align with Australian Commission on Safety and Quality in Health Care (ACSQHC) principles for AI safety and governance, emphasising the importance of stewardship oversight, auditability and accountability. More broadly, they reflect recognised challenges in medical AI, including factual reliability, hallucination and overconfidence. 7 , 18 They underscore the need for structured governance frameworks and clinician supervision to ensure accountability and maintain trust before any consideration of clinical use. 6 , 19 , 20 , 21 Future work should include expanded simulation incorporating resistant organism scenarios, complex comorbidities and contextual modifiers such as recent overseas healthcare exposure. Locked version reproducibility testing and structured usability assessment among junior and perioperative clinicians will be necessary to evaluate workflow integration and training requirements. Prospective evaluation in real‐world clinical settings and integration with hospital electronic systems will be essential to define safety, usability and governance requirements prior to deployment. The two clinically significant errors observed, including omission of required anaerobic coverage and failure to redose, mirror recognised human prescribing failures in surgical prophylaxis, particularly relating to redosing compliance and anaerobic coverage oversight. No hallucinated antibiotic classes or grossly inappropriate agent selections were identified. Within this limited simulation, AI error phenotypes therefore resembled known human lapses rather than qualitatively novel risks. Several limitations warrant consideration. This evaluation relied on a small number of simulated surgical scenarios, which likely overestimate performance relative to the complexity and variability of real‐world clinical practice. Inputs were manually constructed and did not reflect incomplete, ambiguous or evolving patient information commonly encountered in clinical workflows. The structured prompts did not incorporate rare comorbidities or complex contextual modifiers (e.g. neuromuscular disease affecting antibiotic safety, recent overseas healthcare exposure or prior multi‐drug‐resistant organism colonisation) that may materially alter prophylaxis selection. Real‐world deployment would require structured integration of such variables. Harm classification was necessarily hypothetical, and the resulting confidence intervals were wide, indicating substantial imprecision in risk estimates. The model was not fine‐tuned, retrieval‐augmented or integrated with an EMR, limiting operational relevance. Prospective clinical validation will be essential to define real‐world safety before any consideration of clinical use. Importantly, this evaluation was not designed to assess clinical effectiveness and should not be interpreted as evidence supporting clinical deployment. As informal clinician use of generative AI becomes increasingly common, stewardship‐led evaluation of error patterns and safety boundaries is essential to prevent ungoverned adoption. Conclusion This simulation‐based pre‐implementation evaluation demonstrates that guideline‐based AI systems can be configured using publicly available institutional resources but also identifies stewardship relevant safety risks that currently preclude clinical use without further validation. Although overall guideline concordance was high, clinically significant prescribing risks were observed, highlighting that apparent accuracy alone is insufficient as a safety metric for antimicrobial decision support. Simulation‐based safety assessment should therefore be considered a minimum requirement prior to any consideration of deployment of such systems in Australian healthcare settings. As informal clinician use of generative AI continues to increase, stewardship‐led evaluation and governance frameworks will be essential to prevent unregulated adoption and to ensure patient safety. Supporting information Data S1. Supporting Information.