AI/ML Systems Engineering Workbench Framework

This paper presents the framework of a cloud-based Artificial Intelligence (AI) and Machine Learning (ML) workbench that provides services utilization and performance benchmarking. The framework promotes convenience by enabling a centralized platform for software developers and data scientists to perform federated search across various dataset repositories, choose problem domains, like Natural Language Processing, Speech and Computer Vision, and build/validate models. The benchmarking functionality of this framework helps users evaluate and compare performances of various solutions from multiple cloud service providers. The workbench framework consists of two primary layers. The Services layer which is rendered as an AI as a service (AIaaS) model, providing interfaces that connect users to vision, speech and natural language processing (NLP) services from various AI service providers. The Platform layer is an ML as a Service (MLaaS) model providing access to ML model training, tuning, inference and transfer learning tasks that are fulfillable on multiple cloud ML platforms with preset cloud-based compute instances. Benchmarking is provided on the workbench by comparing accuracy metrics on prediction and detection counts, F1 scores and ML training instances setup and completion time. By utilizing these performance benchmarking features, this framework can assist AI and ML practitioners in making informed judgments when selecting a cloud provider for specific activities. Additionally, it will increase the effectiveness and efficiency of data science training for both teachers and students.