A Computational-Efficient Deformable Convolution Network Accelerator via Hardware and Algorithm Co-Optimization

Deformable convolution networks (DCNs) shows performance boosts on object recognition tasks by enabling variable geometric modeling. However, the irregular addressing of memory accesses makes it inefficient for hardware acceleration. In this paper, we propose a computational-efficient hardware accelerator for DCNs. First, a hardware-friendly DCNs inference scheme is introduced based on the original DCNs algorithm with little accuracy loss. Secondly, a hardware accelerator architecture is presented correspondingly, and an speed matching method is introduced to maximizing the number of deformable layers without latency increase. The proposed accelerator is implemented on the Arria 10 FPGA, results of which show that the proposed design achieves the highest throughput and DSP efficiency compared with state-of-the-art DCNs accelerators.