TKIL: Tangent Kernel Optimization for Class Balanced Incremental Learning

When learning multiple tasks in a sequence, deep neural networks tend to loose accuracy on tasks learned in the past while gaining accuracy on the current task. This phenomenon is called catastrophic forgetting. Memory-based Class Incremental Learning (CIL) methods address this problem by re-learning exemplars retained in the memory from previous tasks. However, due to data imbalances between the training data for the current task and the limited exemplars from previous tasks, existing methods struggle to balance the accuracy across all seen tasks. Here, we propose to address data imbalance and in addition to a generic model to learn a set of task-specific parameters. In particular, we propose a novel methodology of Tangent Kernel for Incremental Learning (TKIL) that seeks an equilibrium between current and previous representations. Specifically, TKIL achieves such equilibrium by tuning different task-specific parameters for different tasks with a new Gradient Tangent Kernel (GTK) loss. Therefore, when representing previous tasks, task-specific models are not impacted by the samples of the current task and are able to retain learned representations. As a result, TKIL equally considers the contribution from all task models. The generalized parameters that TKIL obtains allow it to automatically identify which task is being considered and to adapt to it during inference. Extensive experiments on five CIL benchmark datasets with ten incremental learning settings show that TKIL outperforms existing state-of-the-art methods, e.g., achieving 9.4% boost on CIFAR-100 with 25 incremental stages.