PerfML: Smart Management of Complex Performance Data and Analytics

The association between very long response time (VLRT) requests that from latency long tail, and transient resource contention called millibottlenecks, has been established for n-tier web-facing ap-plications, including CPU, memory, and disk. However, how much of latency long tail problem can be explained by millibottlenecks? Using sophisticated performance log data management tools, and teamed ma-chine learning classifiers, this paper shows strong association between hundreds of VLRT request clusters (the latency long tail) and millibot-tlenecks in every kind of server and all their resource types. These promising results suggest that the Millibottleneck Theory may explain most, if not all, latency long tail phenomena.