# Stanford EE292R AI and Data for Science | Autumn 2026 | Multimodal Modeling for Precision Medicine

## Executive summary

This lecture details advanced AI techniques for precision medicine, focusing on multimodal modeling, cross-modal data imputation, and synthetic data generation. The speaker presents work across digital pathology (H&E images, RNA sequencing) and radiology (3D brain MRI, CT scans). Key technical approaches include using Vision Transformers and Variational Autoencoders (VAEs) to impute missing molecular data from routine imaging, and employing conditional diffusion models (including flow matching and reinforcement learning) to generate high-fidelity synthetic medical images for data augmentation and training foundation models.

## Key takeaways

- Multimodal Data Fusion for Prognosis: Combining multiple data types (e.g., multi-omics data, RNA sequencing, histopathology images) using unified deep learning models significantly improves the prediction of patient prognosis compared to single-modality models. This was demonstrated using cohorts like the Cancer Genome Atlas (TCGA) and Stanford cohorts.
- Cross-Modal Imputation (Image to Omics): A Vision Transformer model was developed to impute missing RNA sequencing data (up to 20,000 genes) using high-resolution Whole Slide Images (WSI). This allows routine H&E images to serve as a surrogate for expensive molecular data, enabling patient stratification and drug response prediction.
- Synthetic Image Generation in Radiology: Advanced models, including 3D masked autoencoders and conditional flow matching diffusion transformers, are used to generate synthetic 3D brain MRIs. These models can be conditioned on demographic and disease features (e.g., predicting Alzheimer's progression) to simulate future disease states.
- Reinforcement Learning for Generative Modeling: In CT radiology, the use of Generative Reinforcement Learning (GRPO) post-training significantly improves the model's ability to generate high-fidelity synthetic images with specific findings (e.g., pleural effusion), enhancing the model's predictive accuracy for clinical findings.

## Technical details

- Multimodal Modeling Architectures: Early, late, and intermediate data fusion techniques are used to combine features from diverse sources (e.g., RNA sequencing, histopathology, CT). The goal is to learn a unified embedding that captures the biological context across modalities.
- Digital Pathology Foundation Models: The speaker utilized Vision Transformers and specialized models like Uni (a digital pathology foundation model) to process Whole Slide Images (WSI). The process involves patching the high-resolution images (e.g., 250x250 patches) and clustering the resulting feature vectors to capture tissue heterogeneity.
- Generative Modeling Techniques: Synthetic data generation relies on advanced diffusion models. For pathology, a cascaded diffusion model is conditioned on an RNA-seq feature vector. For radiology, 3D masked autoencoders are used, followed by conditional flow matching diffusion transformers, which condition image generation on demographic and disease features.
- Radiology Representation Learning: Autoencoders (U-Net, Regular AE) are used to learn low-dimensional, robust representations (embeddings) of 3D brain MRIs. The Regular Autoencoder was found to be superior in capturing phenotypic and biological data relevant for downstream tasks like predicting tumor grade or genetic variants.

## Practical implications

- The ability to impute molecular data (RNA-seq) from routine H&E images significantly expands the scope of patient analysis in clinical practice.
- Synthetic data generation allows for data augmentation, particularly for rare diseases or cohorts where data is scarce, improving the training of foundation models.
- The development of robust 3D representation models for brain MRI enables the prediction of complex clinical features (e.g., IDH1 status, brain age) directly from imaging data.
- These methods facilitate *in silico* experiments, such as simulating drug perturbations or tracking disease progression over time.

## Topics

Multimodal AI, Precision Medicine, Foundation Models, Cross-Modal Imputation, Synthetic Data Generation, Digital Pathology, Radiomics, Diffusion Models, Reinforcement Learning (RL), Cancer Genome Atlas (TCGA), GTEx, UK Biobank, ADNI cohort, CPTAC data set

Source: https://www.youtube.com/watch?v=GyjB3GYxEj0