The Multimodal Medical Clustering of Chest X-Ray Images and Clinical Reports Using Deep Visual and Semantic Text Features
-
Humasak Simanjuntak
*
Program Studi Sistem Informasi, Fakultas Informatika dan Teknik Elektro, Institut Teknologi Del, Laguboti,
Indonesia
- Lenni Hutapea Program Studi Sistem Informasi, Fakultas Informatika dan Teknik Elektro, Institut Teknologi Del, Laguboti, Indonesia
- Jonathan Simorangkir Program Studi Sistem Informasi, Fakultas Informatika dan Teknik Elektro, Institut Teknologi Del, Laguboti, Indonesia
- Yolanda Saragih Program Studi Sistem Informasi, Fakultas Informatika dan Teknik Elektro, Institut Teknologi Del, Laguboti, Indonesia
- Jaden Panggabean Program Studi Sistem Informasi, Fakultas Informatika dan Teknik Elektro, Institut Teknologi Del, Laguboti, Indonesia
Abstract
The limited availability of labelled medical data is a real obstacle in building a supervised learning-based Radiology analysis system. This study offers an alternative approach, unlabeled multimodal clustering, that simultaneously combines clinical information sources: chest X-ray images and physician clinical reports. Visual features are extracted from 1,000 images using a pretrained ResNet152, and text features are constructed using a Dual TF-IDF Vectorizer that processed the findings and conclusions columns separately with explicit weightings (0.4 and 0.6), then enriched with 10 related topics from the Latent Dirichlet Allocation (LDA) model. These two modalities are combined through a weighted late fusion strategy after being normalized by L2, with a visual weight of 0.5 and a text weight of 0.5, respectively. Before clustering, dimensionality reduction is performed using UMAP with 50 components and a cosine metric. Determining the optimal K value using the Elbow and Silhouette Score methods in the range of K=2 to K=15 showed K=8 as the best choice with a Silhouette Score value of 0.7524. The clustering results reveal eight clinical groups reflecting diverse diagnostic patterns, ranging from normal to abnormal findings, such as cardiomegaly and pneumonia, that require clinical attention. This approach has the potential to serve as the basis for a medical data exploration system that relies on no label annotations.
References
P. G. Anderson, H. Tarder-Stoll, M. Alpaslan, N. Keathley, D. L. Levin, S. Venkatesh, E. Bartel, S. Sicular, S. Howell , R. V. Lindsey dan R. M. Jones, “Deep learning improves physician accuracy in the comprehensive detection of abnormalities on chest X-rays,” Scientific Reports , vol. 14, no. 1, p. 25151, 2024.
R. Siddiqi dan S. Javaid, “Deep Learning for Pneumonia Detection in Chest X-ray Images: A Comprehensive Survey,” Journal of Imaging, vol. 10, no. 8, p. 176, 23 July 2024.
M. Y. Ouis dan M. A. Akhloufi , “Deep learning for report generation on chest X-ray images,” Computerized Medical Imaging and Graphics, vol. 111, p. 102320, January 2024.
B. D. Katzman, M. Alabousi, N. Islam, N. Zha dan M. N. Patlas, “Deep Learning for Pneumothorax Detection on Chest Radiograph: A Diagnostic Test Accuracy Systematic Review and Meta Analysis,” Canadian Association of Radiologists Journal, vol. 75, no. 3 , p. 525–533, 2024.
B. Oltu, S. Güney, S. E. Yuksel dan B. Dengiz, “Automated classification of chest X-rays: a deep learning approach with attention mechanisms,” BMC Medical Imaging, vol. 25, no. 1, p. 71, 4 March 2025.
T. Dhar, N. Dey, S. Borra dan R. S. Sherratt, “Challenges of deep learning in medical image analysis – improving explainability and trust,” IEEE Transactions on Technology and Society, vol. 4, no. 1, pp. 68-75, 4 January 2023.
F. Gröger, L. Amruthalingam, S. Lionetti, A. A. Navarini, F. Ille dan M. Pouly, “A review and systematic guide to counteracting medical data scarcity for AI applications,” Computer Methods and Programs in Biomedicine Update, vol. 8, p. 100220, September 2025.
Y. Liu, U. R. Acharya dan J. H. Tan, “Preserving privacy in healthcare: A systematic review of deep learning approaches for synthetic data generation,” Computer Methods and Programs in Biomedicine, vol. 260, p. 108571, March 2025.
X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri dan R. M. Summers, “ChestX-ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases,” dalam IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones dan Larson, “CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 1, pp. 590-597, 17 July 2019.
A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark dan S. Horng , “MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports,” Scientific Data, vol. 6, no. 1, p. 317, 12 December 2019.
A. Bustos , A. Pertusa, J.-M. Salinas dan M. D. l. Iglesia-Vayá , “PadChest: A large chest x-ray image dataset with multi-label annotated reports,” Medical Image Analysis, vol. 66, p. 101797, December 2020.
I. Renika, E. H. Pangaribowo dan A. J. Pitoyo, “Exposing Indonesia's Spatial Epidemiology of Pulmonary Tuberculosis: Geographic Pattern Analysis and Cluster Identification, 2000-2023,” Journal of Geosciences and Environmental Studies, vol. 3, no. 1, p. 22, 31 March 2026.
A. Shelke, M. Inamdar, V. Shah, A. Tiwari, A. Hussain, T. Chafekar dan N. Mehendale, “Chest X ray Classification Using Deep Learning for Automated COVID 19 Screenin,” SN Computer Science, vol. 2, no. 300, p. 2021, May 2021.
N. Hayat, K. J. Geras dan F. E. Shamout, “MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images,” dalam In Machine Learning for Healthcare Conference, 2022.
N. Aksoy, S. Sharoff, S. Baser, N. Ravikumar dan A. F. Frangi, “Beyond images: an integrative multi-modal approach to chest x-ray report generation,” Frontiers in Radiology, vol. 4, 15 February 2024.
N. Kaur dan A. Mittal , “RadioBERT: A deep learning-based system for medical report generation from chest X-ray images using contextual embeddings,” Journal of Biomedical Informatics, vol. 135, p. 104220, November 2022.
E. Khan, M. Z. U. Rehman, F. Ahmed, F. A. Alfouzan, N. M. Alzahrani dan J. Ahmad, “Chest X-ray Classification for the Detection of COVID-19 Using Deep Learning Techniques,” Sensors, vol. 22, no. 3, February 2022.
F. D. Adhinata, N. G. Ramadhan, A. Amrulloh dan A. R. Bahtiar, “Comparison of Supervised Learning Methods for COVID-19 Classification on Chest X-Ray Image,” CommIT Journal, vol. 16, no. 2, p. 195–201, 2022.
G. M. M. Alshmrani , Q. Ni , R. Jiang, H. Pervaiz dan N. M. Elshennawy, “A deep learning architecture for multi-class lung diseases classification using chest X-ray (CXR) images,” Alexandria Engineering Journal, vol. 64, pp. 923-935, 1 February 2023.
M. Innat, M. F. Hossain, K. Mader dan A. Z. Kouzani, “A convolutional attention mapping deep neural network for classification and localization of cardiomegaly on chest X rays,” Scientific Reports, vol. 13, p. 6247, 2023.
D. Demner-Fushman , M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez , S. Antani, G. R. Thoma dan C. J. McDonald, “Preparing a collection of radiology examinations for distribution and retrieval,” Journal of the American Medical Informatics Association,, vol. 23, no. 2, p. 304–310, March 2016.
J. Ramos , “Using TF-IDF to Determine Word Relevance in Document Queries,” dalam Proceedings of the first instructional conference on machine learning, 2003.
D. M. Blei, A. Y. Ng dan M. I. Jordan, “Latent Dirichlet Allocation,” Journal of Machine Learning Research, vol. 3 , pp. 993-1022, January 2003.
T. Baltrušaitis, C. Ahuja dan L.-P. Morency, “Multimodal Machine Learning: A Survey and Taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423-443, 2018.
L. McInnes, J. Healy dan J. Melville, “UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction,” arXiv preprint arXiv:1802.03426, 9 February 2018.
E. Becht, L. McInnes, J. Healy, C.-A. Dutertre, . I. W. H. Kwok, L. G. Ng, F. Ginhoux dan E. W. Newell, “Dimensionality reduction for visualizing single-cell data using UMAP,” Nature Biotechnology, vol. 37, no. 1, pp. 38-44, January 2019.
Copyright (c) 2026 Humasak Simanjuntak, Lenni Hutapea, Jonathan Simorangkir, Yolanda Saragih, Jaden Panggabean
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors retain copyright and grant the EXPLORER right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share (copy and redistribute the material in any medium or format) and adapt (remix, transform, and build upon the material) the work for any purpose, even commercially with an acknowledgement of the work's authorship and initial publication in EXPLORER.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in EXPLORER.
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
.png)
