Efficient Neural Networks with Spatial Wise Sparsity Using Unified Importance Map

Exploiting neural network sparsity is one of the most important directions to accelerate CNN executions. Plenty of techniques are proposed to exploit neural network sparsity, where spatial-wise pruning is quite effective for input image. However, previous spatial-wise pruning methods need nontrivial hardware overhead for dynamic execution, due to layer-by-layer binary sampling and online scheduling. This paper proposes a structured configured, spatial-wise pruning technique. Numerous computation will be saved by skipping unimportant region. By using a unified importance map, the computing graph could be compiled in advance to make it more hardware friendly. Additionally, due to multi-level measurement of importance for each region, our method can have a better performance on various tasks. On image classification task, the method can have around 50% fewer top-1 accuracy drop than previous spatialwise pruning methods at similar sparse level. On super resolution and image deraining task, the method can bring $5 \times$ to $19 \times$ acceleration while causing neglectable effect on reconstruction quality. Hardware implementation is also included.