Investigating the effect of variations in the test development process: a case from a safety-critical system

Variation is inherent to a process, and process management demands understanding the nature of variation in quantitative terms, for evaluation and prediction purposes. This understanding requires the identification of process indicators that build the system of variation. To utilize quantitative techniques to understand and improve a software process, more indicators are needed than in a manufacturing process. The need to identify the indicators of a software process and the lack of a generic approach to assess the ability of a software process for quantitative management encouraged us to carry out a sequence of studies that resulted in the development of an Assessment Approach for Quantitative Process Management (A(2)QPM). This paper explains an application of the A(2)QPM to the test development process of an avionics software project and presents the results. The study aimed at understanding the effect of the test design stage and the effect of internal reviews as verification activities in test development, with respect to process productivity and product quality measures. The measurement data collected during the execution of the processes were analyzed by control charts to observe the evidence of process stability. The mean values of measurement data were utilized to make performance comparisons between the various executions of the test development process. The results showed that process productivity was unaffected, but the test procedure quality was positively influenced by the application of test design and internal reviews. The utilization of the A(2)QPM as a guide for the quantitative implementation enabled the systematic evaluation of the test development process and measures prior to analysis. This resulted in the identification of process clusters having stable variation.