Reference-free review-based product question answering evaluation via distant contrastive learning

In E-Commence applications, online shoppers increasingly rely on reviews to find answers for their queries on products. Recently, there has been significant research on product question answering using reviews. While many neural models have been proposed to generate answers from reviews to answer product questions, the evaluation of review-based product question answering (RPQA) models is still largely using lexicon-based text similarity metrics against reference answers. In this paper, we propose a distant supervised contrastive learning network based on modern transformer-based language models by leveraging distant ground-truth community question answers to train a model for RPQA evaluation. At deployment, the learned metric does not require reference answers to evaluate the generated answers for product queries. We conducted extensive experiments on publicly available AmazonQA datasets to evaluate RPQA models. Experiment results show that our learned metric correlates well with human judgements, and outperforms existing lexicon-based or embedding-based metrics.