Ensemble Prototype Network For Weakly Supervised Temporal Action Localization

Weakly supervised temporal action localization (TAL) aims to localize the action instances in untrimmed videos using only video-level action labels. Without snippet-level labels, this task should be hard to distinguish all snippets with accurate action/background categories. The main difficulties are the large variations brought by the unconstraint background snippets and multiple subactions in action snippets. The existing prototype model focuses on describing snippets by covering them with clusters (defined as prototypes). In this work, we argue that the clustered prototype covering snippets with simple variations still suffers from the misclassification of the snippets with large variations. We propose an ensemble prototype network (EPNet), which ensembles prototypes learned with consensus-aware clustering. The network stacks a consensus prototype learning (CPL) module and an ensemble snippet weight learning (ESWL) module as one stage and extends one stage to multiple stages in an ensemble learning way. The CPL module learns the consensus matrix by estimating the similarity of clustering labels between two successive clustering generations. The consensus matrix optimizes the clustering to learn consensus prototypes, which can predict the snippets with consensus labels. The ESWL module estimates the weights of the misclassified snippets using the snippet-level loss. The weights update the posterior probabilities of the snippets in the clustering to learn prototypes in the next stage. We use multiple stages to learn multiple prototypes, which can cover the snippets with large variations for accurate snippet classification. Extensive experiments show that our method achieves the state-of-the-art weakly supervised TAL methods on two benchmark datasets, that is, THUMOS’14, ActivityNet v1.2, and ActivityNet v1.3 datasets.

I. Introduction

Temporal action localization (TAL), aiming at localizing the action instances in an untrimmed video [1], has been applied in many applications, that is, video summarization [2], video retrieval [3], [4], and video surveillance [5]. Unlike fully supervised TAL, which needs time-consuming frame-level annotations, weakly supervised TAL only needs video-level annotations, which is much easier for building large-scale datasets. To tackle this problem, many methods employ the multiple instance learning framework [1] that uses a labeled bag consisting of multiple unlabeled instances (e.g., video snippets). They learn the action classification scores of video snippets to form temporal class activation sequences (CAS) [6], [7]. The CAS is aggregated with a top-k mean strategy to predict the video-level labels. The action classifier in CAS still relies on the unlabeled snippets, which may fail to distinguish the highly similar action/background snippets resulting in incomplete action localization. To distinguish these unlabeled snippets, some methods use unsupervised clustering to learn class-wise prototypes [8], [9]. Each prototype uses a central feature and a variation to cover a cluster of action/background snippets. However, the action snippets contain multiple subactions and their features have large intraclass variations. The background snippets contain many unconstrained frames and their features have complex variations. As shown in Fig. 1(b), the complex variations of snippets hinder the accurate classification of action and background categories. The above methods focus on learning the snippet representation with unlabeled snippets but neglect to alleviate the misclassified prediction caused by the complex variations. It is still challenging to learn the snippet representation using complex variation snippets in weakly supervised TAL. Fig. 1.

Methodological pipeline of our ensemble prototype learning: (a) action snippets and background snippets with large variations, (b) clustered prototype learning, (c) CPL, and (d) ESWL. In this work, we ensemble consensus prototypes of multiple stages for weakly supervised TAL.