ShuffleVITNet: Mobile-Friendly Vision Transformer With Less-Memory

Traditional Convolutional Neural Networks (CNNs) can efficiently acquire local features, while ViT uses a Transformer structure that captures global contextual information. In this paper, a new high-performance lightweight target detector is designed by combining the respective advantages of CNN and ViT: ShuffleViTNet. In this paper, we improve the VIT Block in MobileVIT, firstly, we use channel-by-channel convolution to replace the normal convolution to reduce the memory access cost (MAC), number of parameters (Parmas) and floating-point operations (FLOPs) in the model inference. The 1×1 convolution is then used to fuse the local features learned by the convolution with the global features learned by the Transofmer, which further lightens the model and allows for easier modification of the number of channels and migration to other lightweight neural networks. Finally a lightweight network friendly to mobile devices is constructed using the improved VIT Block. On the ImageNet-1k dataset, ShuffleViTNet achieves 75.2% Top-1 accuracy, the model’s floating-point operation is 536.5 MFLOPs, which is 44.0% less compared to MobileViT-XS, and the cost of memory accesses for model inference is saved by about 25.6%, and the required memory is saved by about 66.37%. For the object detection task on the MS-COCO dataset, ShuffleViTNet obtains higher mAP accuracy. The experimental results show that ShuffleViTNet is a high-performance lightweight network and can be well applied to downstream tasks.