A study of a video analysis framework using Kafka and spark streaming

As the use of various sensors and cloud computing technologies has spread, many life-log analysis applications for safety services for the elderly and children have been developed. However, it is difficult to perform real-time large data processing in clouds due to the computational complexity of the analysis because efficient deployment schemes of streaming computing components over cloud resources have not been well-investigated. In this study, we propose a video analysis framework that collects videos from multiple cameras and analyzes them using Apache Kafka and Apache Spark Streaming. We first investigate the data transfer performance of Apache Kafka and examine efficient cluster configuration and parameter settings. We then apply this configuration to the proposed framework and measure the data analysis throughput. The experimental results show that the overall throughput varies depending on the number of broker nodes that store data, the number of topic partitions of data, and the number of nodes that conduct analysis processing. In addition, it is confirmed that the number of cores is needed to consider for the efficient cluster configuration, and that the network bandwidth between the nodes becomes a bottleneck as the amount of data and the number of components increase.