MoRE: Multi-Modal contrastive pre-training with transformers on X-Rays, ECGs, and radiology/cardiology report
dc.contributor.advisor | Le, Wei | |
dc.contributor.advisor | Tavanapong, Wallapak | |
dc.contributor.advisor | Quinn, Christopher | |
dc.contributor.author | Thapa, Samrajya | |
dc.contributor.department | Department of Computer Science | |
dc.date.accessioned | 2024-10-15T22:28:31Z | |
dc.date.available | 2024-10-15T22:28:31Z | |
dc.date.issued | 2024-08 | |
dc.date.updated | 2024-10-15T22:28:32Z | |
dc.description.abstract | In this thesis, we introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs), and radiology/cardiology Reports. This approach leverages transformers to encode these diverse modalities into a unified representation space, aiming to enhance diagnostic accuracy and facilitate comprehensive patient assessments. To the best of our knowledge, this is the first work to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report. By utilizing contrastive loss, MoRE effectively aligns modality-specific features into a coherent embedding, which supports various downstream tasks such as zero-shot classification and multimodal retrieval. Employing our proposed methodology, we achieve state-of-the-art (SOTA) on the Mimic-IV, CheXpert, Edema Severity, and PtbXl downstream datasets, surpassing existing multimodal approaches. Our proposed framework shows significant improvements in capturing intricate inter-modal relationships and its robustness in medical diagnosis that sets a precedent for future research in multimodal learning in the healthcare sector. | |
dc.format.mimetype | ||
dc.identifier.doi | https://doi.org/10.31274/td-20250502-185 | |
dc.identifier.uri | https://dr.lib.iastate.edu/handle/20.500.12876/5w5pgj5z | |
dc.language.iso | en | |
dc.language.rfc3066 | en | |
dc.subject.disciplines | Artificial intelligence | en_US |
dc.subject.keywords | Artificial Intelligence | en_US |
dc.subject.keywords | Cardiology | en_US |
dc.subject.keywords | Healthcare | en_US |
dc.subject.keywords | Interpretability | en_US |
dc.subject.keywords | Multimodal | en_US |
dc.subject.keywords | Radiology | en_US |
dc.title | MoRE: Multi-Modal contrastive pre-training with transformers on X-Rays, ECGs, and radiology/cardiology report | |
dc.type | thesis | en_US |
dc.type.genre | thesis | en_US |
dspace.entity.type | Publication | |
relation.isOrgUnitOfPublication | f7be4eb9-d1d0-4081-859b-b15cee251456 | |
thesis.degree.discipline | Artificial intelligence | en_US |
thesis.degree.grantor | Iowa State University | en_US |
thesis.degree.level | thesis | $ |
thesis.degree.name | Master of Science | en_US |
File
Original bundle
1 - 1 of 1
No Thumbnail Available
- Name:
- Thapa_iastate_0097M_21743.pdf
- Size:
- 7.01 MB
- Format:
- Adobe Portable Document Format
- Description:
License bundle
1 - 1 of 1
No Thumbnail Available
- Name:
- license.txt
- Size:
- 0 B
- Format:
- Item-specific license agreed upon to submission
- Description: