An Industrial Experience Report on the Adoption of History-based Test Case Prioritization

Many test case prioritization techniques have been proposed with the ultimate goal of speeding up failure detection. In particular, prioritizing based on history has proven to be an effective strategy. Examining the effectiveness of history-based prioritization in the context of manual testing is important because manual testing continues to be widely adopted in industry, despite all the popularity of automated approaches. However, most of the empirical research on this topic has focused on the context of automation testing. In the industrial context where we conducted our study we proposed two prioritization heuristics based on failure history, and we analyzed these heuristics in the context of manual testing. For the evaluation, massive data was collected from seven years of historical information. The results of our experiments using historical real test execution data showed that the effectiveness of the proposed approach was not far from a theoretical optimal prioritization and that they were significantly better than alternative test suite orderings, including the order suggested by a test management tool and the execution order followed by testers during the real execution of the test suites evaluated as part of our study.