Skeleton-Based Human Activity Recognition by Spatio-Temporal Representation and Convolutional Neural Networks with application to Cyber Physical Systems with Human in the Loop

The developments in Cyber-physical systems where smart computers that can sense and understand human behavior will have enormous societal and economic impact facilitating various services in critical infrastructure and everyday life. CPS with Human in the Loop is a system that takes human response into consideration and human presence and behavior are key parts of the system. So these architectures of smart devices will need to interpret human action in real-time and predict humans' immediate intention in complex, noisy and cluttered environments. One of the main goals of researchers will be the development of CPS that can understand complex human activities. In this paper we propose a novel skeleton-based approach utilizing spatio-temporal information and convolutional neural networks for classification of human activities.

I. Introduction

With the advances in computer vision, ubiquitous computing, machine learning and internet of things specifically in the area of sensor technology, it has become possible to analyze human behavior via various ways and at different levels of interaction and interpretation. Understanding human behavior automatically opens up possibilities for new development in human-computer interaction, with a potential of endowing machines with a capacity to attribute meaning to users' behaviour, preferences, personality, emotional state, etc.