TTH-RNN: Tensor-Train Hierarchical Recurrent Neural Network for Video Summarization

Although a recurrent neural network (RNN) has achieved tremendous advances in video summarization, there are still some problems remaining to be addressed. In this article, we focus on two intractable problems when applying an RNN to video summarization: first the extremely large feature-to-hidden matrices. Since video features are usually in a high-dimensional space, it leads to extremely large feature-to-hidden mapping matrices in the RNN model, which increases the training difficulty. Second, the deficiency in long-range temporal dependence exploration. Most videos contain thousands of frames at least, which is such a long sequence that traditional RNNs cannot deal well with. Facing the abovementioned two problems, we develop a tensor-train hierarchical recurrent neural network (TTH-RNN) for the video summarization task. It contains a tensor-train embedding layer to avert the large feature-to-hidden matrices, together with a hierarchical structure of an RNN to explore the long-range temporal dependence among video frames. Practically, the experimental results on four benchmark datasets, including SumMe, TVsum, MED, and VTW, have demonstrated the excellent performance of a TTH-RNN in video summarization.

I. Introduction

As one of the most important data formats, there has been a steep growth of video data in recent years. According to the statistics, the total durations of videos uploaded to the YouTube website exceed 300 h/min [1]. The explosion of video data increases the difficulty for the viewer to browse them efficiently, and puts forward an urgent demand for automatic video analysis techniques [2], [3]. Video summarization is one of the techniques rising for this challenge.