What and How: Generalized Lifelong Spectral Clustering via Dual Memory

Spectral clustering (SC) has become one of the most widely-adopted clustering algorithms, and been successfully applied into various applications. We in this work explore the problem of spectral clustering in a lifelong learning framework termed as
G
eneralized
L
ife
l
ong
S
pectral
C
lustering (GL
$^2$
SC). Different from most current studies, which concentrate on a fixed spectral clustering task set and cannot efficiently incorporate a new clustering task, the goal of our work is to establish a generalized model for new spectral clustering tasks by “What” and “How” to lifelong learn from past tasks. In respect of “what to lifelong learn”, our GL
$^2$
SC framework contains a dual memory mechanism with a deep orthogonal factorization manner: an orthogonal basis memory stores hidden and hierarchical clustering centers among learned tasks, and a feature embedding memory captures deep manifold representation common across multiple related tasks. When learning a new clustering task, the intuition here for “how to lifelong learn” is that GL
$^2$
SC can transfer intrinsic knowledge from dual memory mechanism to obtain task-specific encoding matrix. Then the encoding matrix can redefine the dual memory over time to provide maximal benefits when learning future tasks, and reversely maximize performance for past tasks. To achieve this, we propose an alternative optimization formulation with convergence guarantee for solving our GL
$^2$
SC model. To the end, empirical comparisons on several benchmark datasets show the effectiveness of our GL
$^2$
SC, in comparison with several state-of-the-art clustering models.