A Comprehensive Evaluation Framework of Software Visualizations Effectiveness

Visualizations are useful in dealing with complex software systems, especially in maintenance and evolution tasks. Software visualization tools can help reduce the cognitive burden on practitioners when trying to understand these systems. However, a major challenge in designing new visualization techniques and tools is evaluating their
effectiveness
for specific tasks and users. If a visualization tool is not effective for practitioners, they are unlikely to adopt it. Existing evaluation frameworks for visualizations mainly focus on
expressiveness
, which refers to the ability of the visualization to show all necessary information. However, evaluating the effectiveness of visualizations is an open research problem, especially in terms of quantifying it. To address this problem, we propose a multi-dimensional evaluation framework that focuses on evaluating visualizations in terms of their qualitative, quantitative, and cognitive aspects. The framework includes seven main dimensions and twenty-eight features, with the effectiveness dimension being further subdivided into four sub-dimensions. We validate our framework by using it to evaluate a number of software visualization tools. This validation demonstrates that the framework can be applied to design and evaluate new software visualization techniques and tools.