MoRE: Multi-Modal contrastive pre-training with transformers on X-Rays, ECGs, and radiology/cardiology report

dc.contributor.advisor Le, Wei
dc.contributor.advisor Tavanapong, Wallapak
dc.contributor.advisor Quinn, Christopher
dc.contributor.author Thapa, Samrajya
dc.contributor.department Department of Computer Science
dc.date.accessioned 2024-10-15T22:28:31Z
dc.date.available 2024-10-15T22:28:31Z
dc.date.issued 2024-08
dc.date.updated 2024-10-15T22:28:32Z
dc.description.abstract In this thesis, we introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs), and radiology/cardiology Reports. This approach leverages transformers to encode these diverse modalities into a unified representation space, aiming to enhance diagnostic accuracy and facilitate comprehensive patient assessments. To the best of our knowledge, this is the first work to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report. By utilizing contrastive loss, MoRE effectively aligns modality-specific features into a coherent embedding, which supports various downstream tasks such as zero-shot classification and multimodal retrieval. Employing our proposed methodology, we achieve state-of-the-art (SOTA) on the Mimic-IV, CheXpert, Edema Severity, and PtbXl downstream datasets, surpassing existing multimodal approaches. Our proposed framework shows significant improvements in capturing intricate inter-modal relationships and its robustness in medical diagnosis that sets a precedent for future research in multimodal learning in the healthcare sector.
dc.format.mimetype PDF
dc.identifier.doi https://doi.org/10.31274/td-20250502-185
dc.identifier.uri https://dr.lib.iastate.edu/handle/20.500.12876/5w5pgj5z
dc.language.iso en
dc.language.rfc3066 en
dc.subject.disciplines Artificial intelligence en_US
dc.subject.keywords Artificial Intelligence en_US
dc.subject.keywords Cardiology en_US
dc.subject.keywords Healthcare en_US
dc.subject.keywords Interpretability en_US
dc.subject.keywords Multimodal en_US
dc.subject.keywords Radiology en_US
dc.title MoRE: Multi-Modal contrastive pre-training with transformers on X-Rays, ECGs, and radiology/cardiology report
dc.type thesis en_US
dc.type.genre thesis en_US
dspace.entity.type Publication
relation.isOrgUnitOfPublication f7be4eb9-d1d0-4081-859b-b15cee251456
thesis.degree.discipline Artificial intelligence en_US
thesis.degree.grantor Iowa State University en_US
thesis.degree.level thesis $
thesis.degree.name Master of Science en_US
File
Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
Thapa_iastate_0097M_21743.pdf
Size:
7.01 MB
Format:
Adobe Portable Document Format
Description:
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
0 B
Format:
Item-specific license agreed upon to submission
Description: