RSNA AI Challenge Models Can Independently Interpret Mammograms

Automating detection in mammograms could help radiologists work more efficiently, improve care quality, and reduce costs. Now, new algorithms have shown excellent performance for detecting breast cancers on mammography images, increasing screening sensitivity while maintaining low recall rates.

These algorithms were submitted for the 2023 Screening Mammography Breast Cancer Detection AI Challenge hosted by the Radiological Society of North America (RSNA, Oak Brook, IL, USA). The challenge aimed to source models capable of enhancing cancer detection in screening mammograms and had over 1,500 participating teams from across the world, using a training dataset of around 11,000 images.

Researchers evaluated 1,537 submitted algorithms on 10,830 single-breast exams confirmed by pathology as positive or negative for cancer. The results, published in Radiology, showed a median specificity of 98.7%, sensitivity of 27.6%, and a recall rate of 1.7%. Combining the top three algorithms increased sensitivity to 60.7%, while ensembling the top ten raised it to 67.8%, approaching average screening radiologist performance in Europe or Australia.

The top-performing algorithms complemented each other by identifying different cancers, with thresholds optimized for positive predictive value and specificity. Sensitivity varied by cancer type, imaging equipment manufacturer, and clinical site. The models generally detected invasive cancers more accurately than noninvasive ones, suggesting areas for future optimization.

Many of the submitted models are open source, providing resources for further research and benchmarking. The results could drive improvement in both experimental and commercial AI tools, with the goal of enhancing breast cancer outcomes worldwide. The research team plans to conduct follow-up studies to compare the top algorithms against commercial products using a larger and more diverse dataset.

Follow-up studies will also test performance on smaller, challenging datasets with strong human reader benchmarks, such as those developed by the UK-based PERFORMS scheme. RSNA continues to run its annual AI Challenge, with this year’s competition focusing on detecting and localizing intracranial aneurysms.

"By releasing the algorithms and a comprehensive imaging dataset to the public, participants provide valuable resources that can drive further research and enable the benchmarking that is required for the effective and safe integration of AI into clinical practice," said Yan Chen, Ph.D., lead author of the study.

Related Links:
RSNA