Maintaining Performance of a Machine Learning System Against Imperfect Retraining

Machine learning systems (MLS) often consist of diverse machine learning models to attain demanding and complex objectives. Despite their advanced functionalities, MLSs encounter the inevitable challenge of performance deterioration caused by distribution changes often referred to as dataset shift. When models encountering dataset shift, retraining machine learning model with new datasets is essential to restore the performance. However, model retraining is not always perfect due to component entanglement, resulting in failures to maintain the required level of performance. To address this issue, this paper investigates the impact of imperfect retraining on the overall performance of MLSs, and accordingly propose two maintenance policies, progressive and conservative retraining policies. We consider an MLS consisting of two sequentially-dependent machine learning models and develop continuous-time Markov chains capturing the dynamics of performance degradation and retraining of machine learning models. The results of parametric sensitivity analysis demonstrate that the progressive retraining policy and conservative retraining policy could provide higher service availability in different conditions.