Ensembles of BERT Models for Ancient Chinese Processing

Ancient Chinese holds a significant place within the rich cultural heritage of China. To aid in the organization and analysis of ancient Chinese texts, various natural language processing tools and methods had been proposed. The advent of pre-trained language models (such as BERT) had further facilitated the automated processing of ancient Chinese. Although these models have demonstrated distinct advantages in processing ancient Chinese, there has been no effort to integrate them in order to further enhance their performance. Therefore, we employed various ensemble learning methods to integrate these single models, namely Chinese BERT, Chinese RoBERTa, SikuBERT and SikuRoBERTa. We then examined how well these models performed in sequence labeling tasks related to ancient Chinese, including word segmentation, Part-of-speech tagging, and named entity recognition. The experimental results indicated the ensemble model consisting of SikuBERT + SikuRoBERTa with the grid search method achieved the best performance. Furthermore, our analysis highlighted the significance of selecting appropriate base models and the quantity for ensemble models to performance.