Rethinking Motivation of Deep Neural Architectures

Nowadays, deep neural architectures have acquired great achievements in many domains, such as image processing and natural language processing. In this paper, we hope to provide new perspectives for the future exploration of novel artificial neural architectures via reviewing the proposal and development of existing architectures. We first roughly divide the influence domain of intrinsic motivations on some common deep neural architectures into three categories: information processing, information transmission and learning strategy. Furthermore, to illustrate how deep neural architectures are motivated and developed, motivation and architecture details of three deep neural networks, namely convolutional neural network (CNN), recurrent neural network (RNN) and generative adversarial network (GAN), are introduced respectively. Moreover, the evolution of these neural architectures are also elaborated in this paper. At last, this review is concluded and several promising research topics about deep neural architectures in the future are discussed.

I. Introduction and Background

Since the concept of artificial intelligence (AI) was proposed in the Dartmouth workshop, AI techniques have given rise to great reforms in various aspects of modern society, such as public opinion analysis in social media, recommender system of e-commerce and intelligent manufacturing in industrial production. In the whole AI community, artificial neural network (ANN) is one class of the most critical algorithms with profound influence. Unlike general networks [1], [2], ANN consists of many connected artificial neurons that model the function of biological neurons. These artificial neurons transmit nonlinear activation through some weighted connections. Then, the network can approximate a specific complicated function via stacking enough such transmission. Despite the significant achievements of ANNs, they are limited to their capacity of tackling data in the raw form [3]. Sophisticated feature engineering requiring considerable expertise is necessary for the network to learn reliable patterns from the raw data. Inspired by the investigations to hierarchical structures of human speech system, some novel network architectures with multiple layers are developed, namely deep neural networks (DNNs). These hierarchical neural architectures are expected to overcome the limitation of conventional ANNs.