Towards a Method for Evaluating Realism of Randomly Generated Models of IT Systems

Informally, an information technology system (ITS) consists of hardware and software infrastructure, such as workstations, servers, LANs, etc., along with their connections and interdependencies. Although most organizations operate an ITS, very few share detailed information about their ITS publicly. Because of this, models of arbitrary IT systems have to be generated either manually or automatically, following well known best practices and domain specific requirements. For applications in cyber security, the generated ITS models should be realistic in a sense that a relevant real-world organization could choose to organize its ITS according to them. Since most such models lack a corresponding real-world ITSs for comparison, their realism cannot be evaluated in the sense of conforming to a real-world ITS. In this paper, we propose a method for evaluating arbitrary models of ITSs for which a corresponding real-world ITS is not available for comparison, relying on several visualizations and a questionnaire. We perform a small-scale survey to validate the method, and present its results. The results are encouraging and show that the basic idea of the proposed method can serve the intended purpose, but also indicate several problems that must be addressed before definite conclusions regarding realism can be made.