ConV-ViT: Feature Fusion-based Detection of Gastrointestinal Abnormalities using CNN and ViT in WCE Images

Vision Transformer (ViT) and its variants have gained significant prominence in computer vision due to their exceptional performance across various tasks. However, ViTs are data-hungry models that require vast amounts of data for training. Given the scarcity of medical data, this paper presents a fine-tuned vision transformer specifically designed for small-size datasets. We fine-tuned the original model using convolutional neural networks (CNNs) to extract both high and low-level features from wireless capsule endoscopy (WCE) images. In this work, we integrate a CNN module into the original ViT to extract features from WCE patches, which are then fused with the original ViT features. The classification is accomplished using a multilayer perceptron (MLP) to categorize images into normal and abnormal categories on the Kvasir Capsule Endoscopy dataset, as well as bleeding or non-bleeding categories on the Red Lesion Endoscopy dataset. The experimental findings substantiate the efficacy of the suggested approach, yielding favorable outcomes in comparison to other state-of-the-art methods.