Stanford EE292R AI and Data for Science | Autumn 2026 | Multimodal Modeling for Precision Medicine
Summary
This lecture details advanced AI techniques for precision medicine, focusing on multimodal modeling, cross-modal data imputation, and synthetic data generation. The speaker presents work across digital pathology (H&E images, RNA sequencing) and radiology (3D brain MRI, CT scans). Key technical approaches include using Vision Transformers and Variational Autoencoders (VAEs) to impute missing molecular data from routine imaging, and employing conditional diffusion models (including flow matching and reinforcement learning) to generate high-fidelity synthetic medical images for data augmentation and training foundation models.
Key takeaways
-
Multimodal Data Fusion for Prognosis
2:00
Combining multiple data types (e.g., multi-omics data, RNA sequencing, histopathology images) using unified deep learning models significantly improves the prediction of patient prognosis compared to single-modality models. This was demonstrated using cohorts like the Cancer Genome Atlas (TCGA) and Stanford cohorts.
-
Cross-Modal Imputation (Image to Omics)
5:50
A Vision Transformer model was developed to impute missing RNA sequencing data (up to 20,000 genes) using high-resolution Whole Slide Images (WSI). This allows routine H&E images to serve as a surrogate for expensive molecular data, enabling patient stratification and drug response prediction.
-
Synthetic Image Generation in Radiology
24:10
Advanced models, including 3D masked autoencoders and conditional flow matching diffusion transformers, are used to generate synthetic 3D brain MRIs. These models can be conditioned on demographic and disease features (e.g., predicting Alzheimer's progression) to simulate future disease states.
-
Reinforcement Learning for Generative Modeling
38:20
In CT radiology, the use of Generative Reinforcement Learning (GRPO) post-training significantly improves the model's ability to generate high-fidelity synthetic images with specific findings (e.g., pleural effusion), enhancing the model's predictive accuracy for clinical findings.
Technical details
-
Multimodal Modeling Architectures
170s
Early, late, and intermediate data fusion techniques are used to combine features from diverse sources (e.g., RNA sequencing, histopathology, CT). The goal is to learn a unified embedding that captures the biological context across modalities.
-
Digital Pathology Foundation Models
380s
The speaker utilized Vision Transformers and specialized models like Uni (a digital pathology foundation model) to process Whole Slide Images (WSI). The process involves patching the high-resolution images (e.g., 250x250 patches) and clustering the resulting feature vectors to capture tissue heterogeneity.
-
Generative Modeling Techniques
1600s
Synthetic data generation relies on advanced diffusion models. For pathology, a cascaded diffusion model is conditioned on an RNA-seq feature vector. For radiology, 3D masked autoencoders are used, followed by conditional flow matching diffusion transformers, which condition image generation on demographic and disease features.
-
Radiology Representation Learning
1550s
Autoencoders (U-Net, Regular AE) are used to learn low-dimensional, robust representations (embeddings) of 3D brain MRIs. The Regular Autoencoder was found to be superior in capturing phenotypic and biological data relevant for downstream tasks like predicting tumor grade or genetic variants.
Mentioned resources
- Cancer Genome Atlas (TCGA)
- Cancer Genome Atlas (TCGA)
- GTEx
- UK Biobank
- ADNI cohort
- CPTAC data set
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.