I. Introduction
Video-based surveillance system [1], [2] has been widely applied in many places such as airports, train stations, banks, etc. The general purpose of a surveillance system is to help detecting and preventing illegal and unwanted activities. Recently, due to the widespread increase of security threats, e.g. terrorism, the use of surveillance technologies has become an essential requirement for organizations and individuals. However, most existing surveillance systems still depend on human operations. That raises the issue of having a surveillance system which effectively supports human operators to make proper decisions.