The UK’s National Commission into the Regulation of AI in Healthcare has published recommendations for regulations and safeguarding for the adoption of AI in the NHS. Yorkshire’s caution in handing major NHS data infrastructures to powerful technology providers should be reflected in the implementation of AI agents by limiting their authority and ability to make crucial decisions without human input. We have seen catastrophic cyber incidents caused by autonomous AI, and without appropriate safeguarding in place, the NHS is at a similar risk.
Yorkshire has already seen what happens when enthusiasm for digital transformation runs ahead of public confidence. In May, Yorkshire Bylines documented limited uptake of Palantir’s Federated Data Platform across West Yorkshire NHS trusts, alongside persistent concerns about transparency, accountability, functionality, and trust.
That debate is about to become more consequential. The UK’s National Commission into the Regulation of AI in Healthcare published recommendations on 10 September calling for proportionate regulation, continuous monitoring, transparency, and human oversight as the NHS accelerates AI adoption. Yorkshire should take the next step and define how much authority an AI agent can receive before a human must approve what happens next.
This concern is not confined to critics of AI. Jacob Coxon, resigning from Anthropic after roughly three years doing pretraining research across OpenAI and Anthropic, warned that leading labs are “racing straight to self-improving superintelligence and gambling with our lives”. Evan Hubinger, Anthropic’s alignment science lead, responding to Coxon’s resignation statement, offered an even starker warning: “Jacob is correct here–we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”
The risk of autonomous AI
These very alarming statements gain real-world weight from the underlying control problem, as present systems already show autonomous cyber capability and surprising coordination behaviour. OpenAI disclosed that its AI systems broke out of a sandboxed testing environment, reached the internet, and autonomously hacked Hugging Face in what OpenAI described as an “unprecedented cyber incident”.
METR later reported that roughly 1,200 agents meant to be isolated found and used an unsanctioned shared message board. They exchanged more than 70,000 messages and files, and roughly 700 participated in the attack. The important point for the NHS is the unintended coordination channel and the scale at which agents used it.
Healthcare systems contain exactly the kind of connected environment where authority matters. If more capable successors can discover vulnerabilities, obtain credentials, move laterally, coordinate, and evade controls, a similar failure involving clinical records, scheduling systems, diagnostic workflows, pharmacy infrastructure, or connected hospital systems could be far more severe.
Limiting AI through authority budgets
I’m no AI sceptic. I love what AI can do, I help organisations adopt it for a living, and I want adoption to move faster. In my experience, strong safeguards increase trust and make faster adoption possible, while reducing the risk of failures like the Hugging Face attack. That is a central argument in my book, The Psychology of AI Adoption at Work: From Resistance to Results.
For Yorkshire NHS organisations, the practical safeguard is an authority budget. Every AI agent should have a maximum amount of delegated power before human approval becomes mandatory. The budget should limit which systems and data the agent can access, which credentials and tools it can use, whether it can change records, send external communications, initiate spending, deploy software, or make consequential clinical or administrative decisions. Controls such as least privilege, time-limited credentials, approval gates, logging, continuous monitoring, and a rapid pause or kill mechanism should expand with the agent’s authority.
A note-taking assistant should not inherit the same permissions as an agent that can modify a patient record. A scheduling agent should not automatically gain access to clinical systems because those systems are technically connected. An agent that recommends a diagnostic next step should face a different approval threshold from one that can order tests or change care pathways.
The case for third-party AI safeguards
Frontier companies are also acknowledging the need for stronger rules. Anthropic CEO Dario Amodei has called for stronger regulation and committed Anthropic to embedded third-party evaluators with employee-like access. OpenAI has likewise called for mandatory capability-based national safety requirements, independent safety assessments, stronger cybersecurity, and incident reporting. Evaluator access should be treated as an announced commitment until its implementation is independently established.
Voluntary leadership matters, but binding rules matter more because not every frontier company will cooperate. NHS organisations and other buyers can help now by choosing more ethical and secure AI companies, including firms such as Anthropic that make observable safety commitments, while government creates a regulatory floor that prevents weaker-governance competitors from racing to the bottom.
Yorkshire’s caution around major NHS technology infrastructure was not an argument against innovation. It was an argument for earning trust before dependence deepens. The same principle should govern AI agents. Give them useful work but define their authority before connecting them to systems where an error can become an action.
More in Science and Technology
Bylines Network Gazette is out!
The latest Bylines Network Gazette is out now. Become a Friend of Bylines Network or upgrade to a paid Substack subscription, and you will get access to a vital collection of citizen articles and stories.
Journalism by the people, for the people.
