Test-based and metric-based evaluation of code generation models for practical question answering

We performed a comparative analysis of code generation model performance with evaluation using common NLP metrics in comparison to a test-based evaluation. The investigation was performed in the context of question answering with code (test-to-code problem) and was aimed at applicability checking both ways for generated code evaluation in a fully automatic manner. We used CodeGen and GPTNeo pretrained models applied to a problem of question answering using Stack Overflow-based corpus (APIzation). For test-based evaluation, industrial test-generation solutions (Machinet, UTBot) were used for providing automatically generated tests. The analysis showed that the performance evaluation based solely on NLP metrics or on tests provides a rather limited assessment of generated code quality. We see the evidence that predictions with both high and low NLP metrics exist that pass and don't pass tests. With the early results of our empirical study being discussed in this paper, we believe that the combination of both approaches may increase possible ways for building, evaluating, and training code generation models.

1 Introduction

Today, the problem of natural language generation (NLG) is considered an open challenge applied to many domains. The recent advances include significant success in the programming domain applied in different scenarios such as code search, code comprehension, clone detection, etc. (see, for example, a comprehensive review [1]). Commonly, such problems work with a mixture of two main subdomains, namely code and natural text. As a result, four categories of tasks appear in this area namely code-to-code, text-to-code, code-to-text, and text-to-text each with a multitude of existing tasks, solutions, benchmarks [12].