Journals & Magazines >IEEE Transactions on Computers >Volume: 67 Issue: 5

On-Chip Communication Network for Efficient Training of Deep Convolutional Networks on Heterogeneous Manycore Systems

Download PDF
Download References
Request Permissions
Save to
Alerts

Abstract:

Convolutional Neural Networks (CNNs) have shown a great deal of success in diverse application domains including computer vision, speech recognition, and natural language...Show More

Metadata

Abstract:

Convolutional Neural Networks (CNNs) have shown a great deal of success in diverse application domains including computer vision, speech recognition, and natural language processing. However, as the size of datasets and the depth of neural network architectures continue to grow, it is imperative to design high-performance and energy-efficient computing hardware for training CNNs. In this paper, we consider the problem of designing specialized CPU-GPU based heterogeneous manycore systems for energy-efficient training of CNNs. It has already been shown that the typical on-chip communication infrastructures employed in conventional CPU-GPU based heterogeneous manycore platforms are unable to handle both CPU and GPU communication requirements efficiently. To address this issue, we first analyze the on-chip traffic patterns that arise from the computational processes associated with training two deep CNN architectures, namely, LeNet and CDBNet, to perform image classification. By leveraging this knowledge, we design a hybrid Network-on-Chip (NoC) architecture, which consists of both wireline and wireless links, to improve the performance of CPU-GPU based heterogeneous manycore platforms running the above-mentioned CNN training workloads. The proposed NoC achieves 1.8× reduction in network latency and improves the network throughput by a factor of 2.2 for training CNNs, when compared to a highly-optimized wireline mesh NoC. For the considered CNN workloads, these network-level improvements translate into 25 percent savings in full-system energy-delay-product (EDP). This demonstrates that the proposed hybrid NoC for heterogeneous manycore architectures is capable of significantly accelerating training of CNNs while remaining energy-efficient.

Published in: IEEE Transactions on Computers ( Volume: 67, Issue: 5, 01 May 2018)

Page(s): 672 - 686

Date of Publication: 27 November 2017

ISSN Information:

DOI: 10.1109/TC.2017.2777863

Funding Agency:

Wonje Choi

Electrical Engineering and Computer Science, Washington State University, Pullman, WA

Wonje Choi received the BS degree in computer engineering from Washington State University, Pullman, Washington, in 2013, where he is currently working towards the PhD degree. His research interests include high performance heterogeneous manycore system design and power management for mobile platform. He is a student member of the IEEE.

Karthi Duraisamy

Electrical Engineering and Computer Science, Washington State University, Pullman, WA

Karthi Duraisamy is working toward the PhD degree in the School of EECS, Washington State University, Pullman, USA. His research interests include wireless network-on-chips, high performance manycore platform design, passive optical networks and congestion-aware standard cell placement. He is a student member of the IEEE.

Ryan Gary Kim

ECE, Carnegie Mellon University, Pittsburgh, PA

Ryan Gary Kim received the PhD degree from the Electrical Engineering and Computer Science Department, Washington State University, Pullman, USA. He is currently working as a postdoctoral researcher at Carnegie Mellon University. His research interests are on the energy efficiency and scalability of manycore systems. He is a member of the IEEE.

Janardhan Rao Doppa

Electrical Engineering and Computer Science, Washington State University, Pullman, WA

Janardhan Rao Doppa is an assistant professor with Washington State University, Pullman. His general research interests include artificial intelligence, machine learning, and electronic design automation. He received an Outstanding Paper Award for his structured prediction work at AAAI conference (2013). He is a member of the IEEE.

Partha Pratim Pande

Electrical Engineering and Computer Science, Washington State University, Pullman, WA

Partha Pratim Pande is a professor and holder of the Boeing Centennial chair in computer engineering in the school of Electrical Engineering and Computer Science, Washington State University, Pullman, USA. His current research interests are novel interconnect architectures for multicore chips, on-chip wireless communication networks. He is a senior member of the IEEE.

Diana Marculescu

ECE, Carnegie Mellon University, Pittsburgh, PA

Diana Marculescu is a professor in the ECE Department at Carnegie Mellon Univ. She has won several best paper awards in top conferences and journals. Her research interests include energy-, reliability-, and variability-aware computing and CAD for non-silicon applications. She is a fellow of the IEEE.

Radu Marculescu

ECE, Carnegie Mellon University, Pittsburgh, PA

Radu Marculescu is a professor in the ECE Department at Carnegie Mellon University. He has received several Best Paper Awards in top conferences and journals covering design automation of integrated systems and embedded systems. His current research focuses on modeling and optimization of embedded and cyber-physical systems. He is a fellow of the IEEE.

Contents

1 Introduction

Deep learning techniques have seen great success in diverse application domains including speech processing, computer vision, and natural language processing [1]. While the fundamental ideas of deep learning have been around since the mid-1980s [2], the two main reasons for their recent success are: 1) the availability of large-scale training data; and 2) advances in computing hardware to efficiently train large-scale neural networks using this data.