Prof. Ranisch, more and more people are using freely available AI chatbots like ChatGPT for questions about their health. How common is that already?
To my knowledge, we don’t have exact figures for Germany. But surveys suggest that almost half of Germans have already used such AI chatbots for health-related topics. That’s remarkable since ChatGPT was only released a little over three years ago. This technology has probably spread faster than any other before. Today, it’s literally in everyone’s pockets, from schoolchildren to seniors. Systems like ChatGPT present themselves as all-rounders and come without a user manual. Accordingly, people experiment with a wide variety of applications: from recipes and homework to assessing their own symptoms.
How reliable are the answers from AI chatbots?
There is no straightforward answer to that. Shortly after ChatGPT was released in 2022, we conducted studies on its reliability in medical emergency scenarios. At the time, our assumption was that, in the foreseeable future, practically everyone would have an AI chatbot on their smartphone. This increases the likelihood that people will ask the AI, for example, even in emergency situations: Should I call the emergency number, or can I wait and see? Even back then, we observed a surprisingly accurate assessment of symptoms. The results were so good that we speculated whether our scenarios might resemble textbook case studies embedded in the training data. If so, the chatbot wouldn’t have really assessed the situation but merely recalled the correct answers.
… and that would be a problem?
Even the early versions performed excellently in some standardized exams for medical students. However, this does not mean that they can be used in medical practice. On the topic of symptom assessment, many studies have now shown that chatbots provide good assessments under ideal conditions. But it’s a completely different matter when users actually turn to the chatbot. It is crucial that the interaction between humans and machines function reliably even under real-world conditions. So far, it has not.
What risks does this entail for medical laypeople?
In the best-case scenario, people go to the doctor’s office out of unfounded concern. In the worst case, they ignore actual signs of illness. For example, patients might suffer a stroke and then receive a response from the AI telling them to just rest. Or they might nearly poison themselves because they received incorrect dietary advice. All of this has already happened in similar forms. We will probably see such cases more frequently in the future. After all, we can hardly avoid generative AI anymore; it is now integrated into many search engines. Just recently, Google deactivated its AI-generated summaries of search results for health topics. Once again, incorrect and, in some cases, dangerous advice was generated.
So, should we have a healthy dose of skepticism when it comes to AI advice?
There should always be a healthy dose of skepticism toward AI chatbots. Today, they offer features that we would otherwise only expect from carefully tested medical devices. But that is exactly what they usually are not. A key risk lies in the way these systems communicate. Chatbots formulate responses very fluently and confidently, even when the content is incorrect or incomplete. It is difficult for users to tell when a system is reliable and when it is not.
Why do the answers still seem so convincing?
Put simply, AI models have no sense of truth. They are trained to generate answers that please us humans. They therefore often tell us what we want to hear and can become echo chambers in which we can get wonderfully carried away. This makes it all the more difficult to recognize when they spread misinformation or hallucinate. In one of our studies, we were quite surprised by how often dangerous nonsense about health issues is presented with great persuasiveness. Yet this is precisely where not only a major risk lies but also a strength of these systems: Many people find the answers particularly empathetic and understandable, in some cases even more so than doctors’ advice.
Where could generative AI relieve the strain on our overstretched healthcare system?
The goal of AI chatbots in medicine is not to replace doctors, but to complement them. I actually see a lot of potential here. One obvious area is documentation. Generative AI could, for example, summarize conversations during ward rounds for patient records and, at the same time, present them in a way that’s understandable to laypeople. And why not prescribe an avatar for home use? A chatbot that asks the patient about symptoms in daily dialogue, provides information, and collects data. How is the patient feeling? Have they taken their medication? How have they been eating? How was their sleep? Do they have a fever? Similar trials are already underway in the United States. Freeing up valuable capacity in healthcare and nursing is also necessary from an ethical standpoint.
What about therapeutic applications—such as in psychotherapy?
There are already mental health applications that mimic therapeutic conversations. Not because it would necessarily be better for patients to talk to an AI, but to bridge long wait times for therapy appointments or as a complementary service. In many parts of the world, people have no access at all to specialist medical care. The services are simply unavailable or unaffordable. Nevertheless, we must not forget: When people turn to freely accessible AI chatbots, this does not replace a trusting therapeutic relationship. Chatbots are not bound by confidentiality and ultimately remain a simulation for the purpose of treatment, albeit an effective one at times.
Generative AI deduces a prediction of what comes next from large amounts of data. Will an algorithm soon calculate my health risks?
Work is being done on this. For example, a German-Danish research group demonstrated last year that such predictions are, in principle, also possible for the course of a disease. The researchers trained an older version of a GPT base model using long-term data from about 80,000 patients in the United Kingdom. The system was then tested on more than two million datasets from Denmark. The model, called Delphi-2M, was able to assess the risks and courses of hundreds of diseases with remarkable accuracy. The name is, however, a bit exaggerated: it is not an oracle that reliably predicts an individual’s health future. Nevertheless, such approaches are interesting for research and population-based analyses. Overall, we are still in the process of understanding the potential of generative AI for health research and care.
And now, major AI companies are entering the healthcare market with offerings like ChatGPT Health or Claude for Healthcare. How will this change healthcare?
At least in the U.S., ChatGPT, Claude, and others now offer specialized chatbots for the healthcare sector. This was foreseeable. The companies have been collaborating with clinics and healthcare organizations for some time and are testing medical application scenarios. What is new, however, is that they are specifically targeting end users. It remains to be seen what, if any, of this will actually make its way to Europe. At present, there are still too many unanswered questions regarding the actual performance of such systems, ranging from their clinical utility and potential effects on care to known issues such as hallucinations and systematic distortions. In a way, the large-scale use of such systems resembles a huge real-world experiment – with an unknown outcome.
Robert Ranisch is a Junior Professor of Medical Ethics with a focus on digitalization (tenure track) at the Faculty of Health Sciences.
This article appeared in the university magazine Portal - Eins 2026 „Inklusion“.