Learning Geometric Information via Transformer Network for Key-Points Based Motion Segmentation

With the emergence of Vision Transformers, attention-based modules have demonstrated comparable or superior performance in comparison to CNNs on various vision tasks. However, limited research has been conducted to explore the potential of the self-attention module in learning the global and local geometric information for key-points based motion segmentation. This paper thus presents a new method, named GIET, that utilizes geometric information in the Transformer network for key-points based motion segmentation. Specifically, two novel local geometric information embedding modules are developed in GIET. Unlike the traditional convolution operators which model the local geometric information of key-points within a fixed-size spatial neighbourhood, we develop a Neighbor Embedding Module (NEM) by aggregating the feature maps of k-Nearest Neighbors (k-NN) for each point according to the semantics similarity between the input key-points. NEM not only augments the network’s ability of local feature extraction of the points’ neighborhoods, but also characterizes the semantic affinities between points in the same moving object. Furthermore, to investigate the geometric relationships between the points and each motion, a Centroid Embedding Module (CEM) is devised to aggregate the feature maps of cluster centroids that correspond to the moving objects. CEM can effectively capture the semantic similarity between points and the centroids corresponding to the moving objects. Subsequently, the multi-head self-attention mechanism is exploited to learn the global geometric information of all the key-points using the aggregated feature maps obtained from the two embedding modules. Compared to the convolution operators or self-attention mechanism, the proposed simple Transformer-like architecture can optimally utilize both the local and global geometric properties of the input sparse key-points. Finally, the motion segmentation task is formulated as a subspace clustering problem using the Transformer architecture. The experimental results on three motion segmentation datasets, including KT3DMoSeg, AdelaideRMF, and FBMS, demonstrate that GIET achieves state-of-the-art performance.

I. Introduction

Motion segmentation aims to identify and separate different motion patterns in video sequences, enabling the analysis and understanding of individual object motions or activities. It is an essential task in computer vision and video analysis, with applications in various fields, including vehicle safety [1], surveillance [2], and autonomous driving [3]. Motion segmentation methods can be roughly divided into two categories: pixel-wise and key-points based methods. Pixel-wise motion segmentation [4], [5], [6], [7], which is also referred to as Video Object Segmentation (VOS) [8], assumes dense correspondences of objects over a video as the input and predicts the dense segmentation of moving objects. In contrast, the goal of key-points based motion segmentation is to partition a set of sparse points in the moving objects into individual non-overlapping clusters. This paper focuses on the latter, as we are interested in handling sparse and unordered datasets without any temporal component, which is motivated by the multi-model structures from motions. In recent years, numerous key-points based motion segmentation methods have been developed, but the performance still cannot satisfy the practical requirements due to the complexity of object motions, occlusions, and noises.