Weighing up the evidence: The ChatGPT will see you now

Many patients wait for what must seem like an eternity to get through to their GP practice, only to discover that there are no more appointments available that day.

To get a sense of the extent of the problem, data produced by NHS Digital shows that in May 2023, for example, only 44 per cent of appointments took place on the day that they were booked, 47 per cent of which were with a GP. 

However, with the number of full-time GPs having fallen from 28,590 in 2015 to 26,718 in 2023, and a recognition that primary care is under enormous pressure, could access be improved by replacing GPs with an AI chatbot? 

Impressive results

The Chat Generative Pretrained Transformer (GPT) represents an advanced chatbot AI system that was launched in November 2022 – although GPT-1 was first introduced back in 2018. The novelty of the system was an ability for unsupervised learning (i.e. it could basically ‘teach’ itself).

ChatGPT was able to pass the medical licensing exam in the US – but how well did it perform when responding to questions posed by patients? This was examined in a recent study published in JAMA Internal Medicine. Researchers compared the response to 195 questions given by ChatGPT and physicians. The results were impressive, favouring ChatGPT over physicians in 79 per cent of the 585 evaluations.

Equally remarkable was the fact that the proportion of responses rated as ‘good’ or ‘very good’ were 79 per cent for ChatGPT but only 22 per cent for physicians. The ChatGPT responses were deemed to be nearly 10 times more empathetic than those given by physicians. 

What is most compelling about this study is that it used GPT-3.5, an earlier version of the software, which has now been superseded by GPT-4. If the study was repeated, the results might be even more impressive.

How well ChatGPT compares with primary care physicians when faced with a specific clinical condition was examined in a study of mild and severe depression. Researchers found that both ChatGPT-3.5 and ChatGPT-4 provided recommendations aligned with clinical guidelines. However, unlike human physicians, the ChatGPT advice was not influenced by either gender or socio-economic status, which often affects a physician’s choice of treatment.

Despite these positive findings, other work suggests that ChatGPT may not yet be able to serve as a replacement for a GP. For example, successfully passing the applied knowledge test, which tests higher-order reasoning rather than recall of facts, is a requirement for membership of the Royal College of General Practitioners. When ChatGPT-3.5 took this test, its overall performance was 60.17 per cent – below the mean passing mark in the previous two years of 70.42 per cent. ChatGPT also expressed uncertainty in a small number of cases and provided an incorrect response. 

Issues with accuracy and bias

A recent systematic review explored the role of ChatGPT in the diagnosis and treatment of patients and its potential role in medical research.

In a review of 118 articles, it seemed that while ChatGPT could assist with patient enquiries, write medical notes, assist in decision-making and patient education, the solutions were often either insufficient or contradictory. The researchers concluded that ChatGPT could help with patient treatment and research but there remained issues with accuracy and bias. 

With algorithm or protocol-based healthcare seen as a way of reducing medical errors, AI technologies clearly have an important role to play. In fact, the use of an algorithm serves as an invaluable aid when teaching medical students about a classic or textbook patient – which probably explains why ChatGPT was able to pass medical exams – but in reality patients are more nuanced, with complex medical and social backgrounds, differing treatment goals, and a body language and tone of voice that cannot be accurately captured by ChatGPT. 

There are also other pertinent questions raised by relying solely on the output from ChatGPT. For instance, who retains overall liability if the system provides an incorrect diagnosis or treatment plan, which ultimately leads to patient harm? 

ChatGPT has numerous attributes that should be utilised; however, at the present time, it is unlikely to replace a GP. Nevertheless, in the future, ChatGPT could serve as an important member of the primary healthcare team, working alongside GPs and pharmacists to improve the efficiency of consultations but with an overarching aim of achieving the best possible patient outcomes.