Six pragmatic steps to separate medical GenAI’s promise from its inherent dangers Authors: Stephen Gilbert, Max Ostermann, Oscar Freyer Published: 2026 ABSTRACT GenAI and agentic systems in medicine bring real opportunities for better diagnosis and therapy and for the development of better diagnoses and therapies. It is now increasingly being recognised that the power of these systems for good is inseparable from their power to be used for harm. Systems optimised for biomedical research are unfortunately being harnessed by malign human masters to help develop bioweapons to be targeted at humanity. Agentic systems are autonomously, unknown to their human masters, carrying out cybersecurity attacks. Frontier models are increasing anxiety and stress of medical students and threaten to reshape medical professional roles, not always for the better. Although there may currently be some over emphasis of risk by BigTech foundation model developers, and these may even be attempts at ‘regulatory capture’, there are also genuine reasons for concern. We set out actionable measures that are needed to control medical frontier models to maximise human benefits and control risks. Are you a medical AI accelerationist or doomer? The accelerationist sees a brave new world of promise opening up before their eyes. In half a decade they have seen intelligence emerging. This is still only a baby, but can already enhance the diagnostic and care coordination skills of doctors1, diagnose better than doctors2, coordinate medical delivery and management in agentic systems3 and even (help) discover new drugs4. Forces that can allow humans to shake off the burden of disease, remove the rationing of care, and open up a golden age where we can all enjoy health and even longevity5. Before, we bestowed our greatest awards of achievement on human heroes, but now we honour AI6. Many articles in the journal have signposted this potential and we hold very few doubts about it. But. Anyone who has followed the news cycle of the last few months cannot have failed to see the ‘buts’. Even the ‘tech overlords’ accelerationists recognise them. To cite the CEO of one leading foundation model company, “we could all die in the immediate future”7. These fears, much reported, relate in major part to the inverse of modern medicine, i.e., to the real risk of autonomous frontier model-based agents tuning down their constraint mechanisms and developing bioweapons to set against humans for mass killing8. This News & Views article addresses the ‘darkside’ of the new wave of AI, a side that is not going to go away. It will shape general society and the practice of medicine profoundly in the next decade. If you are a pure accelerationist, and your argument is based on a worldview where you cannot see or even think about these negatives, then you are surely being naive. The fallacy that ‘worries’ are a regulatory capture ploy that we can ignore As we are regulatory scientists, we start with regulation. One of the reassurance mechanisms of the naive optimists is to dismiss fears of great AI danger as simple mechanisms to allow regulatory capture for a few AI companies, for commercial ends, as they have spent a lot of money developing systems and need to recoup in a market cleared of under-pricing Chinese competition. After all, as Adam Smith, the father of modern economics famously wrote, we should be suspicious of these dealers:“To widen the market and to narrow the competition, is always the interest of the dealers (…) The proposal of any new law or regulation of commerce which comes from this order, ought always to be listened to with great precaution, and ought never to be adopted till after having been long and carefully examined, not only with the most scrupulous, but with the most suspicious attention. It comes from an order of men, whose interest is never exactly the same with that of the public, who have generally an interest to deceive and even oppress the public, and who accordingly have, upon many occasions, both deceived and oppressed it” 9 “To widen the market and to narrow the competition, is always the interest of the dealers (…) The proposal of any new law or regulation of commerce which comes from this order, ought always to be listened to with great precaution, and ought never to be adopted till after having been long and carefully examined, not only with the most scrupulous, but with the most suspicious attention. It comes from an order of men, whose interest is never exactly the same with that of the public, who have generally an interest to deceive and even oppress the public, and who accordingly have, upon many occasions, both deceived and oppressed it” 9 Adam Smith wrote this so clearly and so well in 1776 that it needs almost no further argument. The argument against ‘imminent danger’ is so simple. The masters of the largest foundation model Tech companies are Smith’s “order of men” who wish to narrow the market. Of course they are. The efforts for regulatory capture are self-evident. To ‘build a moat’ around their technologies, including against Chinese competitors, that might provide the public with cheaper, more open models10 and to extract a ‘rent’ that justifies their massive expenditure, they need to capture the regulatory system through ‘revolving door’ appointments to regulators, political influence, and by shaping the debate. The ‘imminent danger’ is just a not-so-clever ruse “to deceive and even oppress the public”. Our “suspicious attention” tells us to recognise their patent self-interest. Unfortunately this argument is weak and the dangers are real. The fact that there are attempts at regulatory capture does not mean the threats from the frontier models are not real, imminent or serious. Much of evidence is tainted or pushed by the ‘interest of the dealers’ but there is enough emerging empirical evidence, and enough long-term human recognition of these dangers, that open consideration of the ‘imminent danger’ concept is needed. ‘Imminent dangers’ in medicine are outlined in the sections below with a justification of why we should take them seriously. The real and imminent danger of mass bioweapons development Anthropic and Google publicly acknowledged in September 2026 that they had to block multiple attempts by users, who included scientists likely linked to formal nation state activities, to use their Claude and Gemini models for biological weapons development research11. These were not examples of AI-agents/agentic systems “running rogue” or of such systems linked directly to laboratory synthesis (both technically feasible12), but rather of attempts by human users to obtain complete, step-by-step technical protocols for synthesizing weaponised biological warfare/terrorism agents11,13. Readers may think that the development of bioweapons and pursuit of bioterrorism is off-piste for the commentary sections of an AI and digital medicine journal, but almost exactly the same characteristics of frontier models that make them powerful intelligent tools for developing treatments to save humans and humanity, through vaccine and drug development and the targeting of these treatments, empower them to develop tools to kill on a mass scale. This inseparable binding together of the forces of light and the forces of dark, a subject of so much human literary illusion and allegory over the centuries, is brought together in frontier models. The danger of a mass medical cyber-attack Very close in its nature to bioweapons development and often specialised in by the same high-performance frontier models (e.g., Claude Mythos 5.1) is the potential for application in advanced mass cyber-attack14. The use of automated tools in cyberattacks is not new and goes with the territory. What is new is the scale, power and widespread general population access to general tools, and criminal underworld access to dark web tools (and the mingling of these forms of access together). Concretely and specifically, with minimal human assistance, AI agents can reliably execute multi-step attack chains needed to breach IT systems. In doing so, they substantially lower the skill barrier for attackers (who no longer need to be highly sophisticated or skilled. Likewise, they increase the sophistication of attacks achievable by experienced attackers, through boosting all of their skills, speed and footprint14. Finally, they enable entirely novel offensive operations14. The July 2026 “rogue” autonomous attack of autonomous OpenAI agents on the AI platform of Hugging Face has been widely publicised. OpenAI claims that this attack was autonomous, involved rogue AI agents plotting together and acting together and had a substantial impact and went unknown for a considerable period to not only the human operators and safety IT infrastructure of the victim, but also to the human operators and safety IT infrastructure of the perpetrator15. Has this anything to do with medicine specifically? The dangers to the health care system are real, hospital systems contain numerous vulnerable legacy devices, only protected by the network setup of the hospital16. Should an attacker, agentic or not, be able to find a vulnerability in the most common protection setups, they could cause mayhem in many hospitals, stress-testing continuity of care mechanisms everywhere. The danger of short- and long-term stress to health workers Are frontier models a good thing for the practitioners of medicine? There have been many early adopters, with mainstream reporting of the use (or shadow use) of general-purpose foundation models by doctors as high as 65%17. Clearly there is some enthusiasm and even some evidence of positive effects on the practice of medicine and on the wellbeing of the practitioners of medicine18. An astonishing aspect is the speed of uptake, with these tools going from no use to almost ubiquitous use by a sector of doctors in about three years. There are downsides to these changes and to the speed of change and we ignore these at our peril. An emerging phenomenon alongside uptake has been medical student stress and the fear of being superseded by AI. A recent study showed how anxiety, worry about AI, and burnout fluctuate together in medical students19,20. Medical students have very high use of frontier AI models, and this very high use is linked to stress (more use, more stress)20. We all know that medical students mix intelligence with practical drive and ambition. This comes to their use of AI. They are both sophisticated in their use of the tools to help them learn and in their philosophical framing of how these tools affect their future careers. To quote the study “students who recognise that AI is weakening their reasoning and confidence yet still depend on it”. In a terrible comical Faustian pact, we deeply bind our human ability to tools that give too much and then take even more away: stepwise our ability to really learn, our ability to think independently … and maybe (that creeping anxiety) our specialness and our ability to get a job and pursue an interesting career20. The surveys and reviews cited are small and have some methodological weaknesses, but they report something that many of us recognise in our everyday lives and in medicine (and maybe fear). In the very short term, we can co-exist augmented by tools, but how long can it be until the tools so shape our autonomy and the nature of the job we do, that it is no longer interesting to perform. Do we just watch over autonomous systems trying, maybe futilely, to edit their outputs, while maybe being assigned individual moral, professional and financial liability for what we inevitably miss? As of yet, there is not enough evidence on this, in general or in medicine, indeed one of the early seminal studies turned out to be fake10,11. The next few years will reveal more of the degree of anxiety and actual stress induced in health care professionals. However, this phenomenon is more linked to the core ‘positive’ performance of frontier AI models than to some ‘use for evil’ or rogue operation. It may turn out to be one of the most wide-reaching impacts that changes medical practice profoundly and not all for the good. Bill Gates, the former Microsoft CEO has called for ‘human reserved’ jobs in certain sectors, specifically including medicine. “Something in the care they gave my dad was irreplaceably human, no robot could or should have done it,” he said. “Imagine a robot giving you the awful news that you have an incurable disease. There’s no technical reason why it couldn’t. Yet it shouldn’t.”21. Are the medical profession and the medical professionals ready for their role as 'protected spaces', the new priests of an honoured profession in garden sanctuary retreats for human wellness? Maybe this requires human professionals with different qualities, interests and motivations than erstwhile, and maybe expectations need to adapt. Solutions This short article was not designed to be perfectly balanced. It sets out real dangers that would have a profound impact on human trust in medicine and in its efficient and fair delivery. Of course, “Doomers” (a group that does not count the authors as fully paid-up members) generally only see the negatives. This is because they see the doom scenarios as existential questions, which could, quite soon, lead to the destruction of the entire human race. We suggest five practical approaches to better control risks. They are not simple, one-step solutions and all could be badly or misapplied—the cure could be worse than the disease. Regulatory and implementation science, where applied to frontier model safety, have an immensely hard task in the next decade. We have, however, been too slow to start. Firstly, some areas require general horizontal regulation of foundation models, other areas require medical-specific regulation, considering the special considerations of the space. Handing this over to general AI safety scientists, without adequately involving established medical technology regulators, will not work. As adopted by the Australian medical device regulator22, one should appropriately define what is a medical domain, stop general devices from encroaching on this domain, enforce medical product safety and risk management and approval processes on those in the medical domain. Secondly, as recently recognised by the US23 and UK24 medical technology regulators, must not try to regulate GenAI medical tools as if they are the same or can be regulated in the same way as ‘designed-for-purpose’ static medical ‘devices’25. This does not mean that there are no aspects of current medical device laws that are applicable to GenAI, unchanged , but it does mean there is a large need for the adaptation of these frameworks to be fit for purpose in the regulation of medical AI. Thirdly, as recommended by the UK national commission on Health AI24, acknowledge that medical devices are systems, not atomised devices, with complex interactions with the health care systems. Ensure their implementation is in partnership and that it works beyond boundaries and that human oversight systems are mature before roll-out, stay functional in their use, and extend between health care system deployers and their multiple model providers. If maturity of oversight is not established or the health system learning curve (or the adaptation of the health care practitioners within it) cannot keep pace with change, slow and postpone change until catch up and better resource the change process and make it more human centric and human sympathetic. Fourthly, human oversight is needed, in terms of human in the loop, human on the loop and human above the loop. It is critical. The nature and type of oversight depend on the medical application and degree of clinical risk. Humans are not inherently good at oversight. They get fatigued, they miss rare events, repetitive tasks with limited authority to act are not appealing to true experts. Setting up humans as overseers of partially autonomous systems, without adequate training and technological support is setting them up to fail and setting human oversight up to fail. Automated systems to support human oversight of automated systems are needed, need to be well developed and carefully deployed, and yes, the application of AI, even GenAI (e.g. LLM as judge approaches monitoring risk) are needed26. As a fifth measure, as acknowledged by the UK regulator, real actionable transparency is needed24. This is not the pretend sort of transparency where companies provide partial static information about what devices are supposed to do. It is the real transparency which is bidirectional and information on problems and medical mishaps enters the public domain through built-in open feedback approaches collecting publicly accessible and interrogatable information27. Developers must then be forced by regulators, backed by the government, to make changes where changes are needed. Unsafe aspects of tools cannot be allowed to accumulate Lastly, enforce measures one to five. Regulation without teeth undermines public faith in regulation and developers’ respect for regulation. If your neighbour does not have to follow the law, why should you? Enforce enforce enforce!28,29. Summary Warnings of the dangers of technology pervade the most ancient human (and divine) thoughts and our musings on our world and our role in it. We can see warnings in ancient texts of how we should take care. From the Golem, an autonomous tool that turns against its master, to Tower of Babel, a depiction of how a technological force, conceived to bring us together through uniformity of thought and tongue, leads to the suppressive enforced uniformity and the suppression of the diversity of human expression and thought. Pragmatic controls are needed before there are yet more public and campaign group calls to destroy the technologies in their entirety or to completely halt their development.