Exploring the Word Structure of Ancient Chinese Encoded in BERT Models

In recent years, ancient Chinese processing has gradually become an emerging field. With the advent of pre-trained language models such as BERT, various language models suitable for ancient Chinese have also been developed, which further promotes the development of this field. Although each model had its own characteristics, few studies have analyzed and explored how models “comprehend” ancient Chinese from an internal perspective. Therefore, our study analyzed the features of typical pre-trained language models, including Chinese BERT, Chinese RoBERTa, SikuBERT, and SikuRoBERTa, when encoding the word structure of ancient Chinese. We then correlated these findings with their performance in downstream tasks. The results indicated that these models were sensitive to the distance between tokens (characters or words) when encoding word structures, with attention values decreasing as the distance between tokens increased. Moreover, the Siku models demonstrated higher attention towards the current and surrounding tokens.