We use cookies to understand how you use our site and to improve your experience. This includes personalizing content and advertising. To learn more, click here. By continuing to use our site, you accept our use of cookies. Cookie Policy.

Features Partner Sites Information LinkXpress
Sign In
Advertise with Us

Download Mobile App





Medical AI Models Rely on 'Shortcuts' That Could Lead to Misdiagnosis of COVID-19

By HospiMedica International staff writers
Posted on 25 Jun 2021
Print article
Image: Medical AI Models Rely on `Shortcuts` that Could Lead to Misdiagnosis of COVID-19 (Photo courtesy of National Institutes of Health Clinical Center)
Image: Medical AI Models Rely on `Shortcuts` that Could Lead to Misdiagnosis of COVID-19 (Photo courtesy of National Institutes of Health Clinical Center)
Researchers have discovered that medical artificial intelligence (AI) models rely on “shortcuts” that could lead to misdiagnosis of COVID-19 and other diseases.

Researchers from Paul G. Allen School of Computer Science & Engineering at University of Washington (Seattle, WA, USA) have discovered, AI models - like humans - have a tendency to look for shortcuts. In the case of AI-assisted disease detection, such shortcuts could lead to diagnostic errors if deployed in clinical settings. AI promises to be a powerful tool for improving the speed and accuracy of medical decision-making to improve patient outcomes. From diagnosing disease, to personalizing treatment, to predicting complications from surgery, AI could become as integral to patient care in the future as imaging and laboratory tests are today.

However, when the researchers examined multiple models recently put forward as potential tools for accurately detecting COVID-19 from chest radiography (X-ray), they found that, rather than learning genuine medical pathology, these models rely instead on shortcut learning to draw spurious associations between medically irrelevant factors and disease status. In this case, the models ignored clinically significant indicators in favor of characteristics such as text markers or patient positioning that were specific to each dataset in predicting whether an individual had COVID-19. According to the researchers, shortcut learning is less robust than genuine medical pathology and usually means the model will not generalize well outside of the original setting. Combine that lack of robustness with the typical opacity of AI decision-making, and such a tool could go from potential life-saver to liability.

The lack of transparency is one of the factors that led the researchers to focus on explainable AI techniques for medicine and science. Most AI is regarded as a “black box” - the model is trained on massive data sets and spits out predictions without anyone really knowing precisely how the model came up with a given result. With explainable AI, researchers and practitioners are able to understand, in detail, how various inputs and their weights contributed to a model’s output. The team decided to use these same techniques to evaluate the trustworthiness of models that had recently been touted for what appeared to be their ability to accurately identify cases of COVID-19 from chest radiography.

Despite a number of published papers heralding the results, the researchers suspected that something else may be happening inside the black box that led to the models’ predictions. Specifically, they reasoned that such models would be prone to a condition known as worst-case confounding, owing to the paucity of training data available for such a new disease. Such a scenario increased the likelihood that the models would rely on shortcuts rather than learning the underlying pathology of the disease from the training data.

The team trained multiple deep convolutional neural networks on radiography images from a dataset that replicated the approach used in the published papers. They tested each model’s performance on an internal set of images from that initial dataset that had been withheld from the training data and on a second, external dataset meant to represent new hospital systems. The found that, while the models maintained their high performance when tested on images from the internal dataset, their accuracy was reduced by half on the second, external set — what the researchers referred to as a generalization gap and cited as strong evidence that confounding factors were responsible for the models’ predictive success on the initial dataset. The team then applied explainable AI techniques, including generative adversarial networks (GANs) and saliency maps, to identify which image features were most important in determining the models’ predictions.

When the researchers trained the models on the second dataset, which contained images drawn from a single region and was therefore presumed to be less prone to confounding, this turned out to not be the case; even those models exhibited a corresponding drop in performance when tested on external data. These results upend the conventional wisdom that confounding poses less of an issue when datasets are derived from similar sources — and reveal the extent to which so-called high-performance medical AI systems could exploit undesirable shortcuts rather than the desired signals. Despite the concerns raised by their findings, the researchers believe that it is unlikely that the models they studied have been deployed widely in the clinical setting. While there is evidence that at least one of the faulty models - COVID-Net - was deployed in multiple hospitals, it is unclear whether it was used for clinical purposes or solely for research. According to the team, researchers looking to apply AI to disease detection will need to revamp their approach before such models can be used to make actual treatment decisions for patients.

“A model that relies on shortcuts will often only work in the hospital in which it was developed, so when you take the system to a new hospital, it fails - and that failure can point doctors toward the wrong diagnosis and improper treatment,” explained graduate student and co-lead author Alex DeGrave. “A physician would generally expect a finding of COVID-19 from an X-ray to be based on specific patterns in the image that reflect disease processes. But rather than relying on those patterns, a system using shortcut learning might, for example, judge that someone is elderly and thus infer that they are more likely to have the disease because it is more common in older patients. The shortcut is not wrong per se, but the association is unexpected and not transparent. And that could lead to an inappropriate diagnosis.”

“Our findings point to the importance of applying explainable AI techniques to rigorously audit medical AI systems,” said co-lead author Joseph Janizek. “If you look at a handful of X-rays, the AI system might appear to behave well. Problems only become clear once you look at many images. Until we have methods to more efficiently audit these systems using a greater sample size, a more systematic application of explainable AI could help researchers avoid some of the pitfalls we identified with the COVID-19 models.”


Related Links:
Allen School

Gold Member
12-Channel ECG
CM1200B
Gold Member
Disposable Protective Suit For Medical Use
Disposable Protective Suit For Medical Use
Silver Member
Compact 14-Day Uninterrupted Holter ECG
NR-314P
New
Vein Illuminator
VSI

Print article

Channels

Critical Care

view channel
Image: The permeable wearable electronics developed for long-term biosignal monitoring (Photo courtesy of CityUHK)

Super Permeable Wearable Electronics Enable Long-Term Biosignal Monitoring

Wearable electronics have become integral to enhancing health and fitness by offering continuous tracking of physiological signals over extended periods. This monitoring is crucial for understanding an... Read more

Surgical Techniques

view channel
Image: NTT and Olympus have begun the world\'s first joint demonstration experiment of a cloud endoscopy system (Photo courtesy of Olympus)

Cloud Endoscopy System Enables Real-Time Image Processing on the Cloud

Endoscopes, which are flexible tubes inserted into the body's natural openings for internal examination and biopsy collection, are becoming increasingly vital in medical diagnostics. Their minimal invasiveness... Read more

Patient Care

view channel
Image: The newly-launched solution can transform operating room scheduling and boost utilization rates (Photo courtesy of Fujitsu)

Surgical Capacity Optimization Solution Helps Hospitals Boost OR Utilization

An innovative solution has the capability to transform surgical capacity utilization by targeting the root cause of surgical block time inefficiencies. Fujitsu Limited’s (Tokyo, Japan) Surgical Capacity... Read more

Health IT

view channel
Image: First ever institution-specific model provides significant performance advantage over current population-derived models (Photo courtesy of Mount Sinai)

Machine Learning Model Improves Mortality Risk Prediction for Cardiac Surgery Patients

Machine learning algorithms have been deployed to create predictive models in various medical fields, with some demonstrating improved outcomes compared to their standard-of-care counterparts.... Read more

Point of Care

view channel
Image: The PATHFAST hs-cTnI-II high-sensitivity troponin assay has been developed for the PATHFAST Biomarker Analyzer (Photo courtesy of Polymedco)

POC Myocardial Infarction Test Delivers Results in 17 Minutes

Chest pain is the second leading cause of emergency department (ED) visits by adults in the United States, generating over 7 million visits annually. In the event of a suspected heart attack, physicians... Read more
Copyright © 2000-2024 Globetech Media. All rights reserved.