Expert Evaluation of Export Control-Related Question Answering Capabilities of LLMs

In this paper we introduce evaluation experiments performed by an expert to assess Large Language Models ability to answer questions related to export control in Japanese language. We compare outputs of two popular models, namely ChatGPT and GPT-4, and measure their accuracy in predicting intention of the question and label of the answer. Prediction results indicate that both models were better in guessing question intention but failed to recognize what type of an answer would be most probable. Furthermore, expert evaluation of outputs generated by both models show that most of the answers contain mistakes which can mislead users asking a dialog system for an expertise. We analyze these outputs and enumerate problems to be tackled in the future.