MeLoN: Distributed Deep Learning meets the Big Data Platform

Recent advancements in Artificial Intelligence have brought “Deep Learning” frameworks to be a cornerstone for the 4th Industrial Revolution along with “Big Data” platform technologies such as Apache Hadoop. However, efficient processing of deep learning applications has become challenging as the overall sizes of data and model increase rapidly. To address this problem, we can leverage big data platforms that have successfully provided stable storage and data processing capability during the past decade. In this paper, we present design and implementation of MeLoN (Multi-tenant dEep Learning framework On yarN) that can effectively run distributed deep learning applications on top of the big data platform Hadoop. MeLoN takes expected GPU memory usages of a deep learning application as an input parameter, and employs a GPU over-provisioning policy that can improve the overall resource utilization. Evaluation results show that MeLoN can improve the overall system throughput for concurrently running multiple deep learning applications in a Hadoop cluster. MeLoN can bring many interesting research issues related to profiling of expected GPU memory usages of deep learning applications, storage optimizations for deep learning processing, supporting complex deep learning related jobs based on queuing systems which can ultimately contribute to a new data processing framework in the YARN-based Hadoop ecosystem. In this paper, we present design and implementation of MeLoN (Multi-tenant dEep Learning framework On yarN) that can effectively run distributed deep learning applications on top of the big data platform Hadoop. MeLoN takes expected GPU memory usages of a deep learning application as an input parameter, and employs a GPU over-provisioning policy that can improve the overall resource utilization. Evaluation results show that MeLoN can improve the overall system throughput for concurrently running multiple deep learning applications in a Hadoop cluster. MeLoN can bring many interesting research issues related to profiling of expected GPU memory usages of deep learning applications, storage optimizations for deep learning processing, supporting complex deep learning related jobs based on queuing systems which can ultimately contribute to a new data processing framework in the YARN-based Hadoop ecosystem.