Vision Transformer with Interactive Windows based on Patch Sequence Reconstruction

To further enhance the performance of Transformer-based models in vision task, this paper proposes an algorithm named interactive windows based on patch sequence. Firstly, we introduce a patch internal modeling method that relies solely on convolutional operations to model the internal features of each patch. This not only endows the model with the capability of multi-scale feature representation but also adds translational invariance and inductive bias. Secondly, we propose a sequence reconstruction transformer block, enabling the interaction of attention features between different windows. Finally, we conduct experiments on the ImageNet-1K and COCO 2017 datasets. The experimental results demonstrate that our improved model achieves the best classification accuracy of 87.3% on ImageNet-1K, the best detection accuracy of 59.6% on the COCO dataset, and the best segmentation accuracy of 51.9%.