SIRM: Cost efficient and SLO aware ML prediction on Fog-Cloud Network

Cloud and Fog computing are complementary technologies used for complex Internet of Things (IoT) based deployment of applications. With an increase in the number of internet-connected devices, the volume of data generated and processed at higher speeds has increased substantially. Serving a large amount of data and workloads for predictive decisions in real-time using fog computing without Service-Level Objective (SLO) violation is a challenge. Integration of multiple cloud services and platforms with fog computing can resolve this issue by providing additional resources. In this work, we present a general-purpose System for Inference Request Management (SIRM) aimed at automatically generating a suitable execution workflow to execute ML/DL inference requests using fog with Function-as-a-Service (FaaS) and Machine Learning-as-a-service (MLaaS) offered by cloud vendors. Generated workflow minimizes the cost of deployment as well as SLO violations. The use of SIRM results in less than 5% violations when tested using health domain and recommender system based applications.