On Cost-Sensitive Calibrated Uncertainty in Deep Learning: An application on COVID-19 detection

Reliable and cost-sensitive calibrated estimated uncertainty in deep learning is important in many real-world applications where safety is critical, and prediction problems are asymmetric, in the sense that different types of misclassification errors incur different costs or significant losses which may result in the loss of life in some circumstances. However, uncertainty obtained by approximate inference techniques, such as variational inference, cannot guarantee optimal predictions to represent the model error and is prone to miscalibration (and often poor calibration) due to the assumption of the constant cost of misclassification, which is not realistic in medical diagnosis. Knowing how much confidence there is in a prediction is essential for gaining clinicians' trust in the technology. Bayesian decision theory provides a principled approach for optimal decision making under uncertainty, given a utility function over actions. We propose a variational inference with Monte Carlo Drop-weights based Bayesian neural networks model, which means cost-sensitive calibrated predictive uncertainty can be estimated while minimising asymmetric cost as an expected utility function with improved accuracy. We measured bias-corrected uncertainty using Jackknife resampling technique and propose uncertainty estimation performance metrics, including risk coverage curve, which directly corresponds to well-calibrated estimated uncertainty performance. We have highlighted potential issues in commonly used performance metrics, calibration measures, the quality of the estimated uncertainty and proposed revised metrics to mitigate them. We evaluated the effectiveness of our approach using X-Ray images detecting Covid-19 to improve the reliability of computer-based diagnostics.