Testing the Test: Observations When Assessing Visualization Literacy of Domain Experts

Various standardized tests exist that assess individuals' visualization literacy. Their use can help to draw conclusions from studies. However, it is not taken into account that the test itself can create a pressure situation where participants might fear being exposed and assessed negatively. This is especially problematic when testing domain experts in design studies. We conducted interviews with experts from different domains performing the Mini-VLAT test for visualization literacy to identify potential problems. Our participants reported that the time limit per question, ambiguities in the questions and visualizations, and missing steps in the test procedure mainly had an impact on their performance and content. We discuss possible changes to the test design to address these issues and how such assessment methods could be integrated into existing evaluation procedures.

1 Introduction

Visualization literacy is important-not only to be able to “make meaning from and interpret patterns, trends, and correlations in visual representations of data” [5] but also to be able to discern visualization mirages [26], misleading charts [23], and VisLies [39] that comes with the widespread use of social media [14]. Recently, the visualization community has focused on developing tools and instruments to measure visualization literacy [8], [13], [21], [30]. One example is Mini-VLAT, designed as a fast and effective way of measuring literacy among the general public [30]. With the goal of making the test practical and time-effective, the authors adopted twelve questions from the Visualization Literacy Assessment Test (VLAT) [21] and allocated 25 seconds for each question. While both VLAT and Mini- VLAT were designed to assess the literacy of novice users, in this interdisciplinary work with visualization experts and social scientists, we investigate the applicability and usefulness of Mini-VLAT when working with domain experts. In particular, we conducted structured interviews with seven experts from social science, architecture, engineering, and computer science backgrounds to obtain qualitative insights into the structure, types of questions, and sources of confusion or ambiguities encountered during the test.