LegalDB: Long DistilBERT for Legal Document Classification

Transformers have caused a paradigm shift in tasks related to natural language. From text summarization to classification, these models have established new state-of-the-art results on various general and closed domain tasks. Having said that, most of the popular transformer based models (BERT - Bidirectional Encoder Representations from Transformers, DistilBERT, and RoBERTa) face a limitation in terms of the content length they can accept. This is because the self-attention used by these models scales quadratically with the context sequence length. While this works well for short passages and questions, for longer documents this method fails to effectively capture both the local and global contexts. This shortcoming is underscored further for closed domain tasks like Legal Document classification, where the length of the documents can extend up to a couple of pages and which differ in their vocabulary from general English substantially. In this paper, a new architecture is proposed, where the concept of "long" attention is applied to a distilled BERT and then the model is pre-trained on legal-domain specific corpora. This helps combine a local windowed attention with task-motivated global attention, making the model contextually-aware for longer sequences. The proposed model is also able to outperform fine-tuned BERT and other transformer-based models at the task of legal document classification while also being faster.