Taiwan Legal Longformer: A Longformer-LSTM Model for Effective Legal Case Retrieval

Legal artificial intelligence (LegalAI) has been increasing applications within legal systems, propelled by advancements in natural language processing (NLP). Compared with general documents, legal case documents are typically long text sequences with intrinsic logical structures. However, most existing pre-trained language models (PLMs) struggle with understanding the long-distance dependencies between different sections within the complex structure of legal case documents. Another unique challenge in Taiwan Legal field is the lack of labeled datasets. This deficiency makes it difficult to accurately evaluate model performance. To date, models in Taiwan have yet to be specifically trained on legal case data. Given these challenges, this research aims to develop a model that can effectively understand and process long legal case documents and accurately retrieve similar legal cases. This research proposes a Longformer-LSTM model, Taiwan Legal Longformer, combining Longformer and LSTM in a sequenced manner. The Longformer, pre-trained on a Taiwanese legal case corpus, serves as the base model, processing input documents and generating context embeddings that capture local and global dependencies. These embeddings are then processed by the LSTM layer, adept at capturing long-distance dependencies in sequences. The model in this research is evaluated using our dataset and the Average Entropy of the Reason Clustering (AERC) metric, which is based on the assumption that case scenarios under the same reason of legal cases are similar. Compared with baseline models such as CKIP BERT Base Chinese and Longformer, our experimental results illustrate our model's advancements in handling similarity comparisons within extensive legal judgments.