Goal-Oriented Prioritized Non-Functional Testing with Stakeholders' Priorities

Non-functional testing is crucial in software product line engineering to ensure high-quality end products. However, software complexity, limited testing resources, and time constraints pose challenges in conducting comprehensive testing. Additionally, non-functional testing can be seen as a multi-criteria decision problem since it involves multiple quality attribute requirements. Stakeholders often prioritize certain quality attributes over others based on their specific needs, goals, interests, and the value they perceive each quality aspect holds for them. Therefore, incorporating these priorities during test planning is essential for guiding decision-making and aligning the testing process with their needs. This paper presents a methodology for addressing non-functional testing in goal-driven software product lines. The methodology employs a semi-automated process for non-functional testing that utilizes the domain goal model as a testing basis to identify areas (features) within the application within the test space of quality attribute of the application and employs a multi-criteria decision method for prioritizing those features based on stakeholders' assigned priority values. Our goal-oriented methodology optimizes testing efforts by focusing on critical areas, aligning priorities with stakeholders' expectations to enhance software quality and reliability. The methodology supports three key testing activities: capturing of testable non-functional requirements and use case definition for early testing, effective testing scopes design with stakeholders' priorities, and creation of standardized test cases using pre-designed templates. A prototype toolkit was implemented to support the proposed methodology.