A Unified Acceleration Solution Based on Deformable Network for Image Pixel Processing

Recently, deep neural networks, including convolutional neural networks, deformable convolutional networks, and generative adversarial networks, have shown promising potential for image pixel processing tasks, such as image super-resolution and denoising. These various networks bring different memory accesses and complex computation patterns, challenging their hardware deployments seriously. Therefore, this brief proposes a field-unified method to transform various convolutions into a specific deformable convolution with an adaptive receptive field. Based on the transformation, a tile engine, together with a novel position-decoupling computing flow, is developed to unify the multiple memory access patterns, which maintains the consistency of input/output layout formats. Then, a high-level hardware architecture is presented to flexibly support multiple types of convolutional layers. The proposed design is implemented on a Zynq UltraScale+ FPGA board. Experimental results show that our design obtains around
$1.4\times $
computation efficiency improvement, surpassing existing works in terms of hardware flexibility and computation efficiency significantly.