New Study Highlights Benefits and Risks of AI Explanations in Clinical Diagnosis

As use expands, maintaining accuracy and equity is a growing priority. A new study finds that the benefits and risks of AI explanations vary by user expertise and identifies interface choices that may support safer diagnostic decisions.

Researchers at the Massachusetts Institute of Technology, working with collaborators at Stanford University and Columbia University, evaluated several forms of explainable AI for dermatologic diagnosis. The systems provided either a model prediction with confidence alone, similar-image retrieval to support the prediction, heat maps highlighting influential image regions, or plain-language explanations generated by a large language model. These approaches are intended to make AI reasoning more transparent and help users calibrate trust during image-based diagnosis.

The team tested both nonexperts and clinicians on skin disease recognition with and without these aids. Nonexperts assessed whether mole images were cancerous, while clinicians generated differential diagnoses for dermatologic conditions. The researchers also evaluated a fairness-constrained model designed to reduce bias across skin tones.

All explainability methods improved accuracy among nonexperts, largely because users tended to defer to the AI. Large language model explanations were especially persuasive, increasing confidence even when the model was incorrect, while vague or generic rationales were often judged more convincing. Clinicians were more resistant to incorrect AI guidance and performed best when shown only the model prediction without an explanation. Showing explanations before users formed their own assessment also increased reliance on the AI.

Across tasks, AI outperformed humans when disease presentation was subtle, whereas humans excelled when images contained atypical features or unrelated artifacts. The fairness‑constrained model significantly improved accuracy and reduced diagnostic disparities based on skin tone. The findings, published in Nature Medicine on August 4, 2026, suggest interface designs that require users to commit to an initial hypothesis before viewing AI output may mitigate overreliance.

“Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error. We know that both AI and explainability methods can engage automation bias in humans, and this anchoring effect is something that must be accounted for when we design AI systems,” said Marzyeh Ghassemi, an associate professor in MIT’s Department of Electrical Engineering and Computer Science, a member of the Institute for Medical Engineering and Science, and a principal investigator at the Laboratory for Information and Decision Systems and the Abdul Latif Jameel Clinic for Machine Learning in Health.

“These findings are important as patients increasingly turn to AI to help with their health care. Our findings show that those with the least medical knowledge are most likely to be led astray when explainable AI models give an erroneous output,” said Roxana Daneshjou, a co-author and assistant professor of biomedical data science and dermatology at Stanford University.