It’s taken time for the government to see sense and find an alternative name for misleadingly titled physician associates (PAs) whom the public confuse with qualified GPs. NHS England had planned to treble the number of PAs (which included ‘anaesthesia associates’) to 10,000 by the mid-2030s. But fatal misdiagnoses and errors made by PAs, exposed following a review ordered by Wes Streeting in 2024, has put that plan on hold. Now there’s a new kid on the block. Your GP companion – an AI ‘doctor’.
AI doctors
We’ve had robocop. We’ve had AI that can paint. We’ve had AI agents tell us how to write an email, which sift job applications and exam papers, which miscalculate our bills, create stories, CVs, images, and deepfakes. We’ve even had AI bot, ‘Flynn’, successfully apply and be accepted as an art student at an Austrian university.
As AI innovation grabs the attention of politicians and the media, are we now supposed to put our trust in strings of code inside humanoid bots dubbed AI GPs, AI assistant physicians, robodocs, and AI doctors? Or is a bit of scepticism warranted when many of us have experienced ‘smart’ device failure and ‘the computer says no’ syndrome? After all, we’ve also seen AI bots give inaccurate information, as well as AI companions capable of harmful behaviour including harassment.
AI and robot assisted precision surgery have their place in medicine as adjuncts to human experts. But can we dispense with GPs and their familiar trustable human attributes? Can we instead trust a ‘diagnosis’ given by a string of code via text, voice message, or even a ‘human face’ devised by another string of code? A human face constructed in such a way might lure us to forget that its recommendations are generated by the computer code behind it. It is a trick and a deceit, whose saving grace is its speedy response to our questions.
What’s worth more than speed?
While everyone wants swift responses to health concerns, an answering machine, string of code or algorithm cannot infer the kind of information needed to make appropriate health decisions in the same way that medics can. Verbal, physiological and emotional evidence, as well as real life clinical experience inform medical diagnoses and treatments. Relying solely on an AI ‘doctor’ to interpret such information would be risky. AI agents are far from infallible. Algorithms are biased and medical records and AI agents can be hacked and manipulated. Power outage mid-consultation could also lead to serious errors. And what if Dr AI gets it wrong?
Oops, they did it again
Crucially, AI agents can replicate mistakes, and therefore, reproduce the subsequent consequences. When bad or poor decisions are replicated and form the basis of future decisions, errors expand and risk inducing a sense of mistrust in actual hard, scientific evidence. Facts can be deliberately distorted, myths created, and records forged, faked or destroyed. As the first years of the Covid-19 pandemic showed, health issues are notoriously susceptible to crazy theories and ‘cures’.
Is it safe to ignore how our personal data is being scraped, harvested, cut up, and sold back to us with added errors that AI may have inserted without us knowing? Would we really be as comfortable with Dr AI when Dr AI has been hacked or has ‘misheard’ conversations from our devices or our children’s smart toys, and then ‘the computer says no’ when we try to correct them? AI can and does interact with other AI agents without human intervention. And they can make mistakes, repeatedly compound and embellish them, and present errors as accurate facts.
Commercialising blame
PAs relying on Google or similar make us uneasy. But errors made by Dr AI could be on a different scale when it comes to deciding who is responsible and liable for mistakes. In private health care systems, insurance liability for compensation is a big business. Who or what will decide the locus of blame for diagnostic failures and harms arising from mis-prescribing medication or procedures committed by AI? Will it be another string of code? Or AI agents, qualified doctors or bureaucrats?
And what will be to blame? The data on which Dr AI was ‘trained’ for relying on information scraped from discriminatory, maliciously compromised, error-ridden, outdated, gender or culturally biased sources? What seems ethical to one generation may be deeply distasteful to another so constant human vigilance and critical reflection are vital.
Algorithms are designed to discriminate according to whatever biases or goals are built into the underlying source codes. Rotterdam’s social welfare fraud algorithm is a lesson to all. Building trustworthy and trustable AI is not simple but the risks of not doing so must be addressed at the outset.
Is a cute looking AI doc the answer to labour shortfalls?
AI is being promoted as the answer to workforce shortfalls in the health sector. This is partly the result of extrapolating the convenience, speed, cost savings, and efficiency gains that AI has brought to other sectors, such as car-manufacturing and Amazon warehouses by the replacement of humans by robots. It has been estimated that human-to-human contact may be needed for no more than 20% of customer queries and that AI will resolve the majority of common customer service queries by 2029.
Recently, Tony Blair commented on how AI could improve the efficiency of the government and NHS, stating, “It’s absurd we haven’t yet put all of our NHS data there available to be used for innovation.”
But should minimising the need for human workers be prioritised over human wellbeing?
The question is not whether we should have Dr AIs or not. It is a matter of recognising AI’s limitations from the start and insisting ethical criteria be met. The precautionary principle of not doing harm should be the starting point, not an afterthought in the creation of the underlying algorithms. But the potential for cost cutting and efficiency has yet to be assessed against the potential costs and health consequences.
Perhaps it is not yet time to take Dr AI on trust. Instead, Dr AI should be seen as a tool to be used by a human, not as an alternative to a fully trained, authentic GP.
