Can in-memory/analog accelerators be a silver bullet for energy-efficient inference?

A lot of energy-efficient accelerators for inference has been reported. Our investigation result on dependence of energy efficiency on bit precision of accelerators shows that energy efficiency of in-memory/analog accelerators looks better than that of digital accelerators. However, accuracy of in-memory/analog accelerators is usually deteriorated. In order to practically reduce average bit precision for weights while maintaining accuracy, we introduce our proposed filter-wise quantization technique and a specific HW architecture with variable-bit width MAC units used in a bit-parallel manner. Finally, we discuss whether in-memory/analog accelerators can actually be a silver bullet for energy-efficient inference.