Lightweight Local Transformer for COVID-19 Detection Using Chest CT Scans

Convolutional Neural Networks (CNNs) have mainly failed to explicitly model long-range dependencies, primarily because of their intrinsic locality. To address this issue, Transformers have drawn increasing interest in exploiting long-range dependencies among input data. In this study, we aim to enjoy the merits of both local and global feature extractions in CNN and Transformer architectures. To this end, we go beyond the conventional Transformer frameworks and introduce a highly efficient Transformer architecture for early diagnosis and treatment of COVID-19 patients using CT images. Unlike conventional data-hungry Transformers, our model relaxes the requirement of large-scale training data in vision Transformers and also outperforms the state-of-the-art studies. This flexibility empowers our Transformer architecture to be exploited in data-scarce domains. Moreover, we tailor our Transformer architecture in two ways to embody the principle of locality, which once belonged to CNNs. First, we minimally inject convolutional inductive bias into the early blocks of our Transformer architecture and eliminate standard image patching in the vanilla Transformers. Second, unlike typical patch integration in the standard Transformers, we benefit from a deformable convolution in our architecture to adaptively attend to a small set of key features corresponding to nearby patches. Extensive evaluations verify that our Transformer surpasses its counterparts, alleviates the computational complexity of Transformers, and deals with the lack of large-scale training dataset for COVID-19 diagnosis.