Saeed Hassanpour, PhD

Professor of Biomedical Data Science, Professor of Epidemiology, Dartmouth Geisel School of Medicine

Multimodal AI in Cancer: Measuring What Each Modality

Multimodal AI aims to integrate histopathology, radiology, genomic, and clinical data to improve prediction for individual patients. Reports of multimodal models rarely state two quantities together: what a new modality adds to the data already measured, and whether that gain holds at a new site. This talk examines both in studies across several cancers, organized by four roles that a second modality can play. First, as a label that the image predicts: routine hematoxylin and eosin slides identify a high 21-gene recurrence score in breast cancer with an AUC of 0.87, and 0.83 at an external site, whereas single-gene mutations are predicted with moderate to weak accuracy. Second, as a teacher: a modality available only during training, such as an immunostain, changed the AUC by -0.02 to +0.09 across five tasks with 50 to 211 paired cases. Third, as an input: in five studies, fusion had the highest point estimate, with gains of 0.006 to 0.12 in AUC or concordance index over the best single modality, and the image alone was the best single input in only one study. Fourth, as a tool in practice: the talk describes the external validation, randomized reader study, and regulatory steps that move a model toward clinical use. The conclusion is that a modality is worth what it adds to what is already measured, and only if that gain holds at a new site. The talk closes with open statistical problems for multimodal studies.