Point Estimator Log Tracker for Cloud Monitoring

A cloud management system depends mainly on monitoring systems to perform the right management actions, especially in a high change configuration parameters environment. Monitoring system must provide needed information to cloud manager to describe cloud dynamic state by reading cloud-generated logs and sending them to cloud manager. Log updates should be accurate, instantaneous and sent with minimum time delay. Data sources vary from low to high level of cloud infrastructure resources, or it can be generated from workload demands. Logs are used to discover cloud system status, which is input for future actions in cloud management resources orchestration. A good monitoring system must reduce number of communication transactions with cloud manager and keep a fresh and consistent log update. This work introduces a new method of logs tracking and sampling that can achieve lower logging transactions and cloud system reconfiguration actions, under several types of workload and log data sources. The proposed method, Point Estimator (PE) log tracker, can dynamically adapt to the type of workload providing accurate fresh logs values to cloud manager with minimum number of data transactions.

I. Introduction

Cloud monitoring and log management is important for effective cloud resources orchestration. Real time decision is taken based on the current and old historical system statistics values. Reading logs from cloud logs generator (data sources) is critical to get refreshed and accurate log values. In [1] a Xen hypervisor data sources is studied to get the most accurate with a lower cost log data source for low level logging of Virtual Machine (VM) CPU usage. In [2] a new log-tracing tool is proposed that track logs data sources in a smooth way for KVM hypervisor, which can reduce the logs gathering overhead latency. However for both work, logs must be captured (sampled every periodic time interval) and send to cloud manager. Logs must be captured and filtered or aggregated based on the need of the cloud management system. Cloud manager uses current log values and old traces (historical logs) mainly to make actions on the fly. Cloud manager need to communicate with monitoring tool to get data, ether by pulling the requesting data from monitoring tool or by the monitoring tool continually pushing logs to it. Logging process causes transmission cost over the network and processing cost in cloud management system.