Efficient Parallel Wikipedia Internal Link Extraction for NLP-Assisted Requirements Understanding

Requirements engineering (RE) is a critical set of activities in the software development life cycle (SDLC). Without effective requirements elicitation, organization, communication, and understanding software engineers cannot build quality soft-ware. Thus, it is necessary for software stakeholders to facilitate the SDLC by following best practices and utilizing software tools as needed to ensure requirements are well understood. One area where RE still faces issues, despite stakeholders' best efforts, is the communication of requirements amongst the various stakeholders. Software stakeholders consist of the customers, developers, managers, end users, and others with a vested interest in the software, and they typically all have different skillsets, backgrounds, vernaculars, and understanding of the requirements. These differences naturally lead to miscommunications which can lead to redundant, missing, or conflicting requirements, especially when customer and end user domains include complex vocabularies developers may not be accustomed to, and vice versa, e.g., biology, physics, and medicine. One approach in recent works to address this challenge has been to bridge the communication gap between stakeholders by constructing domain-specific ontologies using natural language processing (NLP) and Wikipedia [1]. With these ontologies, stakeholders have a convenient tool they can use to translate and understand specific requirements in the terminologies they're accustomed to. These techniques have shown promising potential, however there are computational challenges associated with efficiently handling a large dataset like Wikipedia. In particular, parsing internal links from Wikipedia article metadata can be a bottleneck in such ontology-construction systems. In this work we address this issue by implementing a program for memory-efficient parallel internal link extraction from Wikipedia articles. This builds on the work of Rodriguez et al. [2] by optimizing additional phases in the knowledge acquisition process.