X-MAS: Extremely Large-Scale Multi-Modal Sensor Dataset for Outdoor Surveillance in Real Environments

In robotics and computer vision communities, extensive studies have been widely conducted regarding surveillance tasks, including human detection, tracking, and motion recognition with a camera. Additionally, deep learning algorithms are widely utilized in the aforementioned tasks as in other computer vision tasks. Existing public datasets are insufficient to develop learning-based methods that handle various surveillance for outdoor and extreme situations such as harsh weather and low illuminance conditions. Therefore, we introduce a new large-scale outdoor surveillance dataset named eXtremely large-scale Multi-modAl Sensor dataset (X-MAS) containing more than 500,000 image pairs and the first-person view data annotated by well-trained annotators. Moreover, a single pair contains multi-modal data (e.g. an IR image, an RGB image, a thermal image, a depth image, and a LiDAR scan). This is the first large-scale first-person view outdoor multi-modal dataset focusing on surveillance tasks to the best of our knowledge. We present an overview of the proposed dataset with statistics and present methods of exploiting our dataset with deep learning-based algorithms.

I. Introduction

The surveillance robot [1], [2], [3], [4] is being popular topic in robotics field with the development of robotics [5], [6], [7]. Over the past several decades, researchers have studied indoor/outdoor surveillance algorithms applied to multiple fixed cameras [4], [8] and to a mobile robot equipped with a camera [3]. In the recent decade, researchers have started focusing on deep-learning methods to deal with harsh outdoor environments [2], [9], [10], [11], [12]. Moreover, recent methods leverage multi-modal sensors because the RGB camera-only methods for outdoor surveillance have an obvious limitation caused by image quality degradation when handling various environmental changes [9], [13], [14], [15]. Fig. 1.

Examples of the first-person view multi-modal dataset: (a) an RGB image, (b) a thermal image, (c) an IR image, and (d) a night vision image. Our dataset provides mask images (e) corresponding to an RGB image (f). It also provides annotations of bounding boxes and tracking IDs. The dataset has been collected for a long period (’’21) and consists of more than 2.5 million images.

Fig. 2.

Environmental diversity in our dataset: running at night, lying in fog, a walking person with an umbrella in the rain, a fallen person beside a chair in the heavy rain at night with various props (towel, umbrella, hard cap, etc.) and vehicles (car, bicycle, motorcycle, etc.). Our dataset also consists of various places (pavements, roads, warehouses, etc.).