The use of artificial intelligence (AI) models in healthcare has been praised for improving diagnostic decisions in some cases but also criticized for worsening decisions in others. A recent study published in the ‘JAMA’ magazine warns that when AI models absorb biased medical data, they can lead to inaccurate diagnoses, putting patients at risk.
The study, conducted by researchers at the University of Michigan College of Engineering and School of Medicine, focused on the diagnostic accuracy of AI models in patients with acute respiratory failure. They found that while AI models can help improve accuracy in diagnosis, they can also mislead doctors when the models are biased.
According to Sarah Jabbour, a doctoral candidate in computer science and engineering at the University of Michigan, the problem lies in the fact that doctors have to understand the explanations provided by AI models and whether the explanation itself is accurate. The study also found that when doctors were shown biased AI models, their accuracy decreased significantly.
The US Food and Drug Administration (FDA) oversees AI and machine learning-based software used in healthcare and has published guidelines for developers to ensure the safety and reliability of AI models. However, the study suggests that there is still a need for better explanation tools to help clinicians understand why a model makes certain decisions and to identify and mitigate biases in the data used to train the models.
The researchers hope that the study will stimulate further research into the safe application of AI-based models in healthcare and help in the development of better explanation tools. The ultimate goal is to ensure that AI models in healthcare lead to improved outcomes for all demographic groups and to provide medical education around AI and bias.