ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code

In recent years, with the widespread attention of academia and industry on the application of large language models (LLMs) to code-related tasks, an increasing number of large code models (LCMs) have been proposed and corresponding evaluation benchmarks have continually emerged. Although existing evaluation benchmarks are helpful for comparing different LCMs, they may not reflect the performance of LCMs in various development scenarios. Specifically, they might evaluate model performance in only one type of scenario (e.g., code generation or code completion), whereas real development contexts are diverse and may involve multiple tasks such as code generation, code completion, API recommendation, and test function generation. Additionally, the questions may not originate from actual development practices, failing to capture the programming challenges faced by developers during the development processTo address the aforementioned issues, we propose ComplexCodeEval, a new benchmark for evaluating the performance of LCMs in various development scenarios. ComplexCodeEval includes 3,897 Java samples from 1,055 high-star GitHub repositories and 7,184 Python samples from 2,107 high-star repositories. Each function sample in ComplexCodeEval contains multiple annotations (e.g., function signatures, docstrings and reference APIs) to accommodate various downstream tasks. Furthermore, to better reflect diverse development scenarios, each function sample is required to originate from a repository that depends on at least one selected library (based on popularity), and each function sample must invoke at least one API from the selected library. Additionally, each function sample has multiple timestamps to avoid data leakage. Based on ComplexCodeEval, we evaluate the performance of ten LCMs across four tasks (i.e., code generation, code completion, API recommendation, and test case generation) to explore their performance in complex development environments. Furthermore, we conduct an in-depth analysis of the impact of context and data leakage on model performance. Our experimental results reveal several key findings. For instance, LCMs exhibit varying performance across different coding tasks. Additionally, rich contextual information can greatly enhance the performance of LCMs. Moreover, using leaked data for evaluation may lead to an overestimation of model performance, resulting in inaccurate evaluation outcomes that deviate from the performance in practiceCCS Concepts• Software and its engineering → Automatic programming.

1 Introduction

Code has become an important application area for LLMs [3], [4], [7], [13], [21], [22], [24], [25], [29], [30], [34], [37], leading to the emergence of numerous LCMs such as Codex [7], Copilot [2], and CodeLlama [30]. LCMs have been widely adopted for various tasks, including code generation, code completion, test case generation, and API recommendation. This widespread adoption of LCMs has remarkably advanced practical development by not only automating repetitive tasks, but also improving the quality of code and accelerating the overall software development process.