Differentiating data collection for cloud environment monitoring

In a growing number of information processing applications, data takes the form of continuous data streams rather than traditional stored databases. Monitoring systems that seek to provide monitoring services in cloud environment must be prepared to deal gracefully with huge data collections without compromising system performance. In this paper, we show that by using a concept of urgent data, our system can shorten the response time for most ‘urgent’ queries while guarantee lower bandwidth consumption. We argue that monitoring data can be treated differently. Some data capture critical system events; the arrival of these data will significantly influence the monitoring reaction speed which is called urgent data. High speed urgent data collections can help system to react in real time when facing fatal errors. A cloud environment in production, MagicCube, is used as a test bed. Extensive experiments over both real world and synthetic traces show that when using urgent data, monitoring system can lower the response latency compared with existing monitoring approaches.