Assessment of Conflict Structure Recognition and Bias Impact in Japanese LLMs

In recent years, with the evolution and proliferation of large language models (LLMs), their application in various fields, including security, has been expanding. LLMs, which learn from vast amounts of text data and possess the capability to process diverse information, may incorporate biases during the learning process. This necessitates cautious consideration when using LLMs in the realm of security. This study examines the perceptions and biases that Japanese LLMs hold towards global conflict structures, including the warring nations of Ukraine and Russia. We conducted experiments based on a novel method and evaluation metric utilizing sentiment analysis on tweet data. The results revealed that while LLMs have some understanding of conflict structures, they tend to exhibit a positive bias towards specific countries in relationships where the conflict structure is unclear. Furthermore, continual pre-training with Japanese text data showed a tendency to intensify certain biases. These findings highlight the risks associated with using LLMs in the security sector and propose the creation of bias-aware datasets and suggestions for enhancing robustness to ensure fairness towards nations.