MoRE: Multi-Modal contrastive pre-training with transformers on X-Rays, ECGs, and radiology/cardiology report
Date
2024-08
Authors
Thapa, Samrajya
Major Professor
Advisor
Le, Wei
Tavanapong, Wallapak
Quinn, Christopher
Committee Member
Journal Title
Journal ISSN
Volume Title
Publisher
Altmetrics
Abstract
In this thesis, we introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs), and radiology/cardiology Reports. This approach leverages transformers to encode these diverse modalities into a unified representation space, aiming to enhance diagnostic accuracy and facilitate comprehensive patient assessments. To the best of our knowledge, this is the first work to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report. By utilizing contrastive loss, MoRE effectively aligns modality-specific features into a coherent embedding, which supports various downstream tasks such as zero-shot classification and multimodal retrieval. Employing our proposed methodology, we achieve state-of-the-art (SOTA) on the Mimic-IV, CheXpert, Edema Severity, and PtbXl downstream datasets, surpassing existing multimodal approaches. Our proposed framework shows significant improvements in capturing intricate inter-modal relationships and its robustness in medical diagnosis that sets a precedent for future research in multimodal learning in the healthcare sector.
Series Number
Journal Issue
Is Version Of
Versions
Series
Academic or Administrative Unit
Type
thesis