LAMPT: LAbel Mask-Predicted Transformer for Extreme Multi-label Text Classification

Extreme Multi-label text Classification ( XMC) is a task of recalling the most relevant labels for each given text from an extremely large-scale label set. It is emphasized that XMC is a more complex classification task because there are two main problems: large space of labels and the labels in XMC tasks tend to be correlated. Existing methods attempt to model label correlations by viewing the XMC task as a sequence generation problem however they still suffer from (1) using the slow serial decoding strategy where labels are predicted one-by-one. (2) needing to compare a mass of label ordering strategies in the decoding stage to achieve satisfied accuracy. In this work, we propose LAbel Mask-Predicted Transformer (LAMPT) to address the both issues, which is a novel non-autoregressive generation model that (1) enriches the input raw text representation with the additional label features by fully exploiting the label dependencies, (2) allows for efficient parallel decoding thanks to its non-autoregressive decoding formulation and mask-prediced training strategy. Experimental results demonstrate that single model performance is substantially enhanced by LAMPT. On a Wiki dataset with thirty-one thousand labels, LAMPT-XLNet accuracy has gained 1.6% relative improvement on P@3 over the LightXML-XLNet. Also, the P@1 of ensemble LAMPT is 90.00%, a significant i mprovement over the state-of-the-art ensemble LightXML (transformer-based) and AttentionXML (LSTM-based), which achieve 89.45% and 87.47%, respectively.

I. Introduction

Extreme Multi-label text Classification (XMC) is an essential task in natural language processing (NLP), which aims to recall multiple relevant labels and discriminate them from enormous numbers of irrelevant labels for a given text. This kind of problems is prevalent in many real-world applications, such as news annotation [25] and recommending related queries on a search engine [1].