Confidence in Prognostic Performance Indicators: How much Data is Enough?

With more reliance on prognostic methods, there remains a gap in understanding the data requirements and accordingly, the model performance. The relative lack or abundance of historical examples dictates the overall confidence in the metrics reported by any prognostic strategy. These metrics, in turn, determine the overall operational value of the prognostic. Sparse data can be used to trick operators into believing the numbers without the knowledge of the uncertainty around those numbers. This work examines the relationship between the quantity of data and the prognostic performance confidence interval. The number of historical examples drives the sampling uncertainty, adding to the detection uncertainty. The combined uncertainty governs both the detection success (Recall) and the amount of life sacrificed in order to avoid the unscheduled failure event. Both of these entities are fundamental to justifying the prognostic strategy, and therefore the amount of historical data is a key factor. As the estimated deterioration progresses, there is an operational tradeoff between earlier removal and event detection. In this manner, the sampling error contribution to overall prediction uncertainty has an operational impact.