Interpretable Transformers for Alzheimer Disease Diagnosis on Multi-modal Data

Alzheimer’s disease (AD), a neurological condition, primarily affects brain cells. Historically, the detection of this disease has focused on single modality data using machine learning and artificial intelligence algorithms. However, recent advancements in machine learning have allowed for the analysis of multi-modal data sources and input types, thereby enhancing the ability to predict Alzheimer’s disease. This research introduces an innovative approach to enhance Alzheimer’s disease research by utilizing a multi-modal data-specific method that applies transformers to both image and text data. In the initial stage, the challenge of data pre-processing for multi-modal datasets is addressed through the use of a U-net based segmentation technique, effectively isolating the Region of Interests (ROIs) in MRI images. The second stage involves the deployment of vision transformers (ViT) and BERT to process the pre-processed data. This application is crucial in handling the complexities associated with multi-modal datasets, particularly those that combine image and textual information. Lastly, our method prioritizes data explainability and interpretability by incorporating advanced Explainable AI (XAI) techniques, specifically Local Interpretable Model-Agnostic Explanations (LIME) and Layer-wise Relevance Propagation (LRP). These techniques provide a deeper understanding of the model’s decision-making process, enabling a more comprehensive interpretation of the multi-modal datasets. We used medical demographic and image data of Alzheimer’s patients from Kaggle for our study. Our proposed method achieved an accuracy of 86%, outperforming other methods.