Systems Monitoring and Big Data Analysis Using the Elasticsearch System

Modern applications produce large quantities of data in the form of logs and events in order to facilitate quick failure diagnosis and mitigation. Special Big Data database systems like Elasticsearch are needed to store and manage these logs. Through such a system, the data is used to query different events and trace down issues that appear in the application. To further improve the efficiency of failure discovery and prevention, the query mechanisms can be extended with Machine Learning techniques to observe anomalies in the infrastructure and automatically alert the managers or administrators. This research project is tracking the development of a scalable Elasticsearch monitoring system. In later stages, the focus will fall on enhancing the system's detection functionality by applying Machine Learning jobs on the stored data.