MFS: A Brain-Inspired Memory Formation System for GAN

Generative adversarial networks (GANs) are subject to catastrophic forgetting when learning stream of data. Inspired by the knowledge of neuroscience, this article develops a memory formation system (MFS) to establish memory for GANs. MFS is composed of three modules, including the identifier, weights upgrade (WU), and weights reactivate (WR). These modules simulate the function of encoding, consolidating, and retrieving in memory formation of human. Identifier creates indexes to label continuous tasks and these indexes are used as a cue when the corresponding tasks are recalled. WU and WR work as synaptic consolidation and system consolidation, respectively. In WU, a novel method, weight saliency measure (WSM) is proposed to measure the saliency of weight. Valuable weights are protected and invaluable weights are released for update when GAN is trained for the new task. In WR, traditional data-replay methods are improved by selecting
${k}$
representatives in each class and their feature vectors are restored. When pseudo data of this class are regenerated, only the samples with distance less than a threshold can be used in future training. Experimental results based on the testing of continual image generation and continual 3-D reconstruction show that MFS can establish a memory system to handle the catastrophic forgetting problem effectively.