Video action re-localization using spatio-temporal correlation

Video re-localization plays an important role in locating the moments of interest in a long videos, and is critical for a variety of applications such as surveillance video monitoring and retrieving similar archived videos for further comparison and analysis. Current re-localization approaches compute a feature vector using a video query for each video frame, and explore various feature matching techniques. These features do not capture information from varying temporal windows, and the dimension reduction to a vector leads to loss of spatio-temporal context. For efficient feature comparison and matching among thousands of videos, we design a Siamese Spatio-Temporal network comprising Convolution Neural Network and Long Short-term Memory blocks (CNN-LSTM) for feature extraction, followed by a correlation layer for spatio-temporal feature matching. We extract video features at varying temporal scales, and localize one or more segments in the reference video that semantically match the query clip. Our approach is evaluated on two benchmark datasets: AVAv2.1- Search and ActivityNet-Search. We show an improvement of over 12% in the mean average precision compared to existing approaches. We perform ablation experiments and show that the modular architecture and the holistic feature extraction expands the scope of this work to multiple video search applications.

1. Introduction

With the rapid advancement in imaging and computing technologies, there is unprecedented growth in the amount of data that we generate every day. To cope with this explosion, proper organization is essential for on-demand retrieval of multi-media information. Visual data is unstructured and bulkier than text. As the database grows, the difficulty of efficient storage and retrieval of relevant search results increases. Traditional search engines index visual data based on the manual annotation of the surrounding metadata such as titles and meta-tags. There are research prototypes using deep learning techniques for automatically extracting details like actions, objects, and captions from videos. The similarity in this metadata can be further used to scrape relevant videos during a search. A downside to this approach is that textual descriptors are often inadequate to describe videos, simply because the same video can be described in different ways. Moreover, retrieval based on such meta-data as query yields too many results, making the search inefficient. With frequent adding and updating of multimedia in massive databases, it is also highly impractical to perform manual entry of all the attributes. Further, manual tagging of industrial videos is in its infancy. Content-based retrieval by using a video example as a query addresses many of these issues. It provides more flexibility and has the ability to query attributes such as texture or shape that are difficult to represent using keywords.