Replicability Study: Corpora For Understanding Simulink Models & Projects

Background: Empirical studies on widely used model-based development tools such as MATLAB/Simulink are limited despite the tools' importance in various industries. Aims: The aim of this paper is to investigate the reproducibility of previous empirical studies that used Simulink model corpora and to evaluate the generalizability of their results to a newer and larger corpus, including a comparison with proprietary models. Method: The study reviews methodologies and data sources employed in prior Simulink model studies and replicates the previous analysis using SLNET. In addition, we propose a heuristic for determining code-generating Simulink models and assess the open-source models' similarity to proprietary models. Results: Our analysis of SLNET confirms and contradicts earlier findings and highlights its potential as a valuable resource for model-based development research. We found that open-source Simulink models follow good modeling practices and contain models comparable in size and properties to proprietary models. We also collected and distribute 208 git repositories with over 9k commits, facilitating studies on model evolution. Conclusions: The replication study offers actionable insights and lessons learned from the reproduction process, including valuable information on the generalizability of research findings based on earlier open-source corpora to the newer and larger SLNET corpus. The study sheds light on noteworthy attributes of SLNET, which is self-contained and redistributable.

I. Introduction

There are only a few empirical studies of open-source MATLAB/Simulink artifacts, maybe due to a widespread perception that open-source Simulink artifacts are typically small, do not represent closed-source development, and are often hard to acquire [14], [15], [27], [61], [70]. Most empirical Simulink studies to date have instead relied on academic-industry collaborations—to get access to large closed-source Simulink artifacts [6]. Most empirical results on Simulink development and artifacts are thus based on case-studies of closed-source artifacts that (even when providing detailed experimental design descriptions and measurement tools) are hard to reproduce or replicate [10].