Study: AI does not prevent misdiagnoses

There are high hopes for AI systems that are supposed to help doctors make decisions. The extent to which so-called Computerized Diagnostic Decision Support Systems (CDDSS) actually help is the subject of controversial debate. A study published in “Lancet Digital Health” now shows that ten percent of diagnoses made by a CDDSS are incorrect. According to the scientists, misdiagnosis is one of the “most common and costly medical problems worldwide”.

Due to the high time pressure, making a diagnosis is a challenge, especially in emergency departments. This is why Inselspital, Bern University Hospital and the University of Bern wanted to find out whether AI could make things easier. According to the researchers, their work is the “world's first study on AI-based diagnostic systems in acute medicine”.

The study examined 1204 patients who were treated in four emergency departments in Switzerland between June 2022 and June 2023 with “non-specific complaints” such as fainting, abdominal pain or fever. 559 of these were examined using the CDDSS “Isabel Pro DDx Generator” from the UK-based company Isabel Healthcare, which, according to researchers, had the highest accuracy at the time.

In the control group, which comprised 645 test subjects, Isabel Pro was not used. The researchers measured the quality of the diagnosis based on how the patients were doing two weeks after treatment, whether they “required unplanned medical follow-up, whether diagnoses were subsequently changed, whether an unexpected intensive care admission was required or whether there were any deaths”, according to the Inselspital.

The study found that the results did not differ between the two test groups. A “diagnostic quality risk” occurred in 18 percent of patients in both groups. “There were also no differences between the groups in terms of serious adverse events and resource consumption, measured in Swiss francs,” say the study leaders.

“No measurable effect” so far

“Currently available AI will not solve the problem of misdiagnosis,” says Prof. Dr. med. Wolf Hautz, Head Physician at the University Department of Emergency Medicine and lead author of the study. “AI-based diagnostic support has no measurable effect for patients in emergency medicine. Regardless of whether you look at medical, economic or procedural differences,” summarizes Hautz.

The study was funded in part by the National Research Program “Digital Transformation” of the Swiss National Science Foundation, which is also supporting the establishment of a working group on “Collaborative Decision-Making”. In the future, the researchers want to evaluate various areas of application for CDDSS. There are also corresponding efforts in Germany. In Lower Saxony, for example, the Research Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed) wants to find out how AI can help with decision-making and thus reduce the burden on the healthcare system.

(mack)