MSDB: A Massive Sensor Data Processing Middleware for HBase

With rapid development and extensive usage of sensors technology, large volumes of sensor data are generated. How to effectively store and process sensor data has been widely concerned. However, with the characteristics of chronological order, large volume and high concurrency, there are a number of difficulties, in particular, hot spot problem, data scatter problem and high concurrent transaction processing. In this paper, we proposed a data processing middleware, namely MSDB, to solve issues discussed above when HBase is used as the data storage medium. A pre-split strategy is designed to deal with hot spot problem and a new design of HBase table is employed to solve data scatter problem. In order to handle high concurrency transactions, a buffer mechanism and a node mapping mechanism are also designed for data caching. Experimental results demonstrate that the MSDB overcomes issues discussed above and performs stably when process massive real scene sensor data using HBase.