Finding Problems: When Digital Library Users Act as Usability Evaluators

Users in digital library usability evaluation typically participate as subjects in studies designed and conducted by usability experts and digital library researchers. What happens however when users take the role of the researchers, and with some basic HCI training, design and conduct their own evaluation of a digital library? For several years, teams of students in master's level HCI classes at Drexel University were given the assignment of designing and carrying out heuristic evaluations of the interface of the Internet Public Library. Their final evaluation reports regularly focused on what, to a usability expert, would not be considered interface issues, such as problems with finding resources in the library. These outcomes contrasted with those of a parallel evaluation of the IPL carried out by doctoral students with a background in HCI, which found interface issues to be the main concerns. A post hoc comparison and analysis of these evaluations highlights differences between users' and evaluators' perceptions of usability, and has implications for the design of digital library evaluation and the roles of users and evaluators in such evaluation.

1. Introduction

Usability evaluation is an important component of digital library development [3], [4]. Data for usability evaluation is collected through a number of methods. Some of these methods involve users participating as subjects in studies designed and conducted by usability experts and digital library researchers. Four underlying assumptions of these studies are: (a) that developers are not typical digital library users (for instance, they are already experts in the design and use of the interface in question) and therefore need to consult users in order to identify requirements for their designs; (b) that users are in possession of useful opinions and knowledge about the usability of a digital library interface that are unknown to developers; (c) that these opinions and knowledge can be elicited from them through the medium and use of properly structured evaluation instruments; and (d) that such methods in general involve a ‘translation’ of users' needs and knowledge (as revealed in the study) into useful knowledge for design requirements.