LLM Diagnostic Toolkit: Evaluating LLMs for Ethical Issues

The rapid proliferation of large language models (LLMs) has brought with it both opportunities and challenges. While LLMs and more broadly generative AI technologies are capable of providing excellent improvements for various routine and autonomous tasks thereby enabling cost and performance benefit, they are also prone to personal and societal harms such as biases, stereotypes, misinformation, and hallucinations to name a few. These ethical concerns have in turn triggered stakeholders across the world to call in for regulatory measures that ensure safe and beneficial use of generative AI technologies. In parallel, there are also research efforts to alleviate these issues through the development of generative AI bias detection and mitigating strategies. Towards advancing this goal, in this paper, we propose an accessible and end-user-friendly LLM diagnostic toolkit whereby diverse stakeholders such as software engineers, business executives, and consumers can examine a suite of LLMs for uncovering a host of ethical issues including biases and misinformation embedded in LLMs. We also demonstrate that our toolkit can be used to diagnose for issues related to commonsense reasoning capabilities of LLMs. Extensive experiments on challenging tasks and datasets demonstrates the effectiveness of our diagnostic toolkit.