Range Based Hardware Optimization of Multilayer Perceptrons with RELUs

To address severely constrained hardware platforms, we propose a range-based hardware inference optimization of Multilayer Perceptrons (MLP). Given a pre-trained MLP model, we simulate the model with training set and collect the values (profile) at the inputs of the RELU activation functions in the hidden layer. If the value range in a neuron is negative then we can prune it as it results in a zero after RELU (note that RELU converts all negative values to 0). Non-negative range neurons can also be pruned provided they degrade the model accuracy minimally. We validate the proposed approach for MLP networks implemented on Virtex®-7vx485tffg1157-1 FPGA with Vivado (version v.2019.2). For three classification datasets and over 10 different trained MLP models per dataset, we obtain average savings of 31.8 % multipliers and 31.8% adders, with average loss of 1.42% in model accuracy.