Collaborative AI System Improves Medical Diagnostic Accuracy

Broad “generalist” models handle many tasks but often lack disease-specific precision. Narrow “specialist” models deliver accuracy yet require task-by-task redevelopment and struggle to converse naturally with clinicians. To help address this challenge, researchers have now developed a collaborative framework that combines generalist and specialist systems to improve diagnostic performance while lowering adaptation costs.

The Generalist–Specialist Collaboration (GSCo) framework was created by a team at The Hong Kong University of Science and Technology (HKUST) in collaboration with international partners. GSCo builds on MedDr, an open-source generalist model trained on more than two million multimodal medical samples. Lightweight specialist models are trained with a single consumer-grade graphics processing unit, enabling local development within hospitals.

During inference, specialist models retrieve similar historical cases and generate diagnostic predictions. MedDr then serves as the decision-maker, integrating those specialist outputs with its embedded medical knowledge to deliver the final diagnosis. This approach aims to pair task-specific expertise with broad reasoning in a way that supports clinical dialogue and reduces brittleness on unfamiliar data.

Validation spanned 32 public datasets comprising about 260,000 medical images across image diagnosis, visual question answering, and radiology report generation. MedDr surpassed prior medical generalist models, while GSCo achieved the highest overall rank in internal and external benchmarks, outperforming 10 state-of-the-art vision foundation models. In skin lesion testing, specialists trained on HAM10000 and validated on the unseen BCN20000 dataset produced a GSCo diagnostic score of 0.8420, exceeding MedDr’s 0.7545 in zero-shot conditions and the specialist model’s 0.8292.

In a human evaluation of chest X-ray report generation, six of seven board-certified radiologists preferred GSCo over the specialist model R2GenGPT. Stress tests also showed MedDr’s resilience as an arbitrator, correctly identifying 67.6% of tumor images despite biased inputs suggesting normal findings. These results indicate a consistent uplift when generalist reasoning is combined with specialist evidence.

Operationally, adapting GSCo to a new clinical task can reduce development costs by up to 100-fold compared with fine-tuning a foundation model. Hospitals can train specialist models on-site and share them anonymously, preserving patient privacy. The team plans to extend MedDr to three-dimensional computed tomography and magnetic resonance imaging, medical video, and more advanced retrieval and test-time reasoning strategies. 

The work was published in Nature Biomedical Engineering with collaborators from Harvard Medical School, Weill Cornell Medicine, The Chinese University of Hong Kong, The University of Hong Kong and Sun Yat-sen University.

“Generalists and specialists in medical AI each possess distinct strengths—one is flexible but imprecise across many diseases, while the other is accurate but narrow, focusing on a single task. The core value of GSCo lies in facilitating their synergistic collaboration, where the generalist model acts as the decision-maker, integrating expert advice with its own broader medical knowledge,” said Chen Hao, assistant professor at The Hong Kong University of Science and Technology and director of the Collaborative Center for Medical and Engineering Innovation.

“We hope this novel framework offers a more practical and sustainable pathway for integrating medical foundation models into everyday clinical workflows, particularly enabling health care settings with limited AI computing resources to share in the benefits of AI-powered medical innovation,” added Prof. Chen Hao.

Related Links
HKUST