Deviant: A Mutation Testing Tool for Solidity Smart Contracts

Solidity is a widely used scripting language for developing smart contracts in blockchain applications. Quality assurance of Solidity contracts is of critical importance because bugs can lead to considerable loss of assets. As software testing is a common practice for quality assurance, many Solidity projects have included built-in tests. It is unclear, however, what level of quality assurance these built-in tests can achieve. This paper presents Deviant, a mutation testing tool for Solidity smart contracts. It automatically generates mutants of a given Solidity project and runs all mutants against the given tests to evaluate their effectiveness. To simulate various faults in Solidity smart contracts, Deviant provides mutation operators for all the unique features of Solidity according to the Solidity fault taxonomy, in addition to the traditional programming constructs. We have used Deviant to evaluate the effectiveness of the tests for three Solidity projects. The results indicate that these tests have not yet achieved high mutation scores and that a test suite adequate for the statement and branch coverage criteria of Solidity smart contracts does not necessarily provide a high-level assurance of code quality. Such observations offer important guidelines for Solidity developers to implement more effective tests in order to deliver trustworthy code and reduce the risk of financial loss.