Multimodal AI Framework Aims to Guide Cancer Immunotherapy Decisions

Standard biomarkers, including programmed death-ligand 1 (PD-L1) expression, microsatellite instability, and tumor mutational burden, provide incomplete and inconsistent guidance across tumor types. Integrating molecular, cellular, imaging, and clinical signals could more fully capture dynamic tumor-immune interactions. A new review in Science Bulletin outlines how multimodal AI can support this integration, presenting a multi-scale framework to guide patient selection, toxicity prediction, and future model evaluation.

Researchers at Army Medical University (Chongqing, China) led the review, “Emerging Advances in Multimodal AI in Cancer Immunotherapy: From Multi-Scale Data Integration to Clinical Decision Support,” which identifies multimodal artificial intelligence as a pathway to unify disparate datasets for clinical decision support. The article describes how models can connect genomics, transcriptomics, proteomics, epigenomics, single-cell and spatial omics, digital pathology, radiological imaging, and longitudinal electronic health records to refine treatment choices and anticipate adverse events. By learning nonlinear relationships across scales, these systems aim to move beyond single biomarkers toward composite patient representations.

Methodologically, the authors map six connected stages: early machine learning classifiers; unsupervised representation learning with autoencoders, variational autoencoders, and contrastive learning; graph-based modeling of cell-cell and molecular interactions; multimodal foundation models and biomedical language models; reinforcement and active learning for dynamic optimization; and “AI virtual cells” that learn cellular states and predict responses to perturbations.

At the molecular and cellular levels, integrated models can characterize cellular composition, functional states, spatial neighborhoods, and communication networks within the tumor immune microenvironment. The review also highlights applications such as AI-assisted virtual staining, cellular phenotyping, and spatial multi-omics integration to maximize information from limited tissue.

At the clinical interface, digital pathology models using H&E whole-slide images and immunohistochemistry have been explored to infer PD-L1 status, microsatellite instability, tumor mutational burden, immune infiltration, treatment response, and prognosis. Radiomics and deep-learning analyses of CT, PET, and MRI can derive phenotypic features associated with response, progression, recurrence, survival, and immune-related adverse events. Longitudinal multimodal analysis may further help distinguish pseudoprogression from true progression or hyperprogression by integrating imaging, pathology, molecular states, and clinical trajectories.

The review emphasizes modeling efficacy and toxicity together to support therapy selection, follow-up planning, and timely intervention. Key barriers include asynchronous data collection, missing modalities, distribution shifts across institutions and devices, shortcut learning, and suboptimal multimodal fusion. Many studies remain retrospective and single-center with limited external validation and variable endpoints, and concerns persist around model opacity, hallucinations, causal interpretation, and insufficient biological validation.

To advance translation, the authors call for multicenter, temporal, and real-world validation, including explicit testing of missing-modality robustness, data drift, calibration, and predictive uncertainty. They also recommend decision-curve analysis and net-benefit assessment to evaluate clinical utility.

The article underscores the need for transparent data provenance, interoperability, privacy protection, and lifecycle monitoring, supported by functional experiments, prospective studies, and clinical trials. AI virtual cells will also require high-quality perturbation datasets, standardized benchmarks, and closed-loop iteration with experiments. Overall, the work positions AI as an assistive tool for organizing cancer immunotherapy around integrated efficacy-toxicity decision support rather than isolated biomarkers.