Adapter-Based Incremental Learning for Face Forgery Detection

Many existing face forgery detection methods primarily revolve around learning general representations on predefined datasets and subsequently crossing these static representations to other datasets. However, these approaches could lead to catastrophic forgetting in real-world scenarios, especially when new forgery methods continually emerge. In this paper, we proposed a novel incremental learning framework for face forgery detection, where we design an adapter-based incremental learning scheme combined with a confidence-based ensemble prediction mechanism. When confronted with new forgery methods, we incorporate small trainable adapter modules, which are retrained along with their corresponding classification layers, yielding a series of task-specific modules. Then we incorporate a confidence-based ensemble prediction mechanism to aggregate all predictions. Through comprehensive evaluations on multiple benchmark datasets (FF++, DFD, and Celeb-DF), our method successfully mitigates the catastrophic forgetting problem in a cost-effective manner and attains state-of-the-art performance in cross-dataset scenario.

1. INTRODUCTION

As advancements in deepfake technologies lead to heightened realism in facial manipulation images [1], [2], the field of face forgery detection has become an increasingly captivating research area. Many existing methods primarily concentrate on within-dataset detection [3] –[5], where forgery methods are predefined and known in both the training and testing sets. Due to the continual emergence of forgery methods in practical scenarios, methods with high within-dataset detection accuracy usually encounter a significant performance degradation in cross-dataset scenario, thereby constraining their broader applicability. When a neural network is retrained to learn from a new dataset, it tends to forget the knowledge it acquired during the initial training on the previous task, a phenomenon referred to as the catastrophic forgetting problem [6].