DARe: DropLayer-Aware Manycore ReRAM architecture for Training Graph Neural Networks

Graph Neural Networks (GNNs) are a variant of Deep Neural Networks (DNNs) operating on graphs. GNNs have attributes of both DNNs and graph computation. However, training GNNs on manycore architectures is a challenging task because it involves heavy communication that bottlenecks performance. DropEdge and Dropout, which we collectively refer to as DropLayer, are regularization techniques that can improve the predictive accuracy of GNNs. Moreover, when implemented on a manycore architecture, DropEdge and Dropout are capable of reducing the on-chip traffic. In this paper, we present a ReRAM-based 3D manycore architecture called DARe, tailored for accelerating on-chip training of GNNs. The key component of the DARe architecture is a Network-on-Chip (NoC) that reduces the amount of communication using DropLayer. The reduced traffic prevents communication hotspots and leads to better performance. We demonstrate that DARe outperforms conventional GPUs by up to 6.7X (5.6X on average) in terms of execution time, while being up to 30X (23X on average) more energy efficient for GNN training.

I. Introduction

Graph Neural Networks (GNNs) are used for predictive analytics using graph-structured data. This makes them different from traditional Deep Neural Networks (DNNs) that operate on regular data structures such as images or sequences. GNN s have various real-life applications such as recommendation systems [1], quantum chemistry [2], social networks [3] [4] etc. To learn representation using the relational structure of graphs, GNN s perform iterative neighborhood aggregation, where each node aggregates features of its neighbors to compute new features [5]. This gives rise to repeated message-passing operations. GNN s exhibit characteristics of both DNN training (involving trainable weights) and graph analytics (accumulating neighboring vertices' information along graph edges). Hence, GNN training is both compute- and communication-intensive.