Functional, Control and Data Flow, and Mutation Testing: Theory and Practice

The growth of user request for higher software quality has motivated the definition of methods and techniques in order to improve the way software is developed. Several works have investigated a variety of testing criteria in an attempt to obtain a testing strategy with lower application costs and higher efficacy in detecting faults. The aim of this chapter is to present the theoretical and practical aspects related to the software testing activity. A synthesis of functional, structural, and fault-based testing techniques is presented. A comparison of the testing criteria (cost, efficacy, and strength) is also considered from the theoretical and experimental points of view. The importance of testing automation is discussed, characterizing the main efforts of the testing community in this direction. Emphasis is given to state-of-practice tools. Efforts from the academia is also discussed. The testing activity and its related problems are presented and illustrated through practical examples with the support of different testing tools, to provide information about software testing in terms of both theory and practice. The need for a systematic evaluation of these criteria and tools under the perspective of experimental software engineering is also discussed. After a short introduction, Section 2 briefly describes a software product named Identifier, which is used to illustrate the testing concepts discussed in this chapter. In Section 3, a synthesis of the functional, structural, and fault-based testing techniques is presented. Sections 4 to 6 provide some examples of different testing criteria, considering the three testing techniques. We comment on their complementarity and show how they can be used in an incremental testing strategy. The importance of testing automation is also discussed in these sections, characterizing the main efforts of the testing community in this direction. In Section 7 the need for a systematic evaluation of these criteria and tools under the perspective of experimental software engineering is discussed. We illustrate that section by describing an experimental evaluation of different test suites generated by random, functional, and Pex [263] (described in detail in Chapter 5) against mutation testing. Finally, Section 8 presents the final considerations of this chapter.