Exploring the Space of Model Comparisons

Deploying machine-learning (ML) models is difficult and fraught with peril. Replacing an old model with a new one may introduce new biases and weaknesses that were easily over-looked. Unlike software updates where we have best practices (unit tests and the like), such best practices for ML are only now evolving. The ML deployment pipeline suffers further from a fracture: on the one side, one has the “data-science” (DS) pipeline, in which one extracts, loads, transforms, and maintains the vast lakes of data the models need to be trained on; on the other side, one has the “ML” pipeline in which experts test and evaluate models, often comparing many, for fitness for the task. To progress ultimately, these two pipelines must be integrated into a single DS/ML pipeline. We posit that doing so rests on model explainability and comparison.