Decoding Logs for Automatic Metric Identification

Automated Log Analysis tasks such as root cause analysis and fault prediction play a pivotal role in maintaining the overall application health. These tasks employ log parsers to extract the dynamic (variable) and constant (template) parts of a log line to generate a template. However, our observations indicate that not all templates carry equal significance. Hence, there is a need to prioritize which templates/variables to use for log analysis. In this paper, we introduce LogMId, a Logs-based Metric Identification method, which is designed to extract critical IT metrics from logs. Through LogMId, we aim to en-hance monitoring, observability tools and in turn Site Reliability Engineers to mine better insights from log data. We showcase the effectiveness of LogMId on a popular log analysis task of anomaly detection. Our experiments indicate that integrating previously used benchmark tools with LogMId features lead to improved results. Additionally, LogMId demonstrates effectiveness even with a smaller amount of training data, emphasising its utility.