Towards More Precise Coincidental Correctness Detection With Deep Semantic Learning

Coincidental correctness (CC) is a situation during the execution of a test case, the buggy entity is executed, but the program behaves correctly as expected. Many automated fault localization (FL) techniques use runtime information to discover the underlying connection between the executed buggy entity and the failing test result. The existence of CC will weaken such connection, mislead the FL algorithms to build inaccurate models, and consequently, decrease the localization accuracy. To alleviate the adverse effect of CC on FL, CC detection techniques have been proposed to identify the possible CC tests via heuristic or machine learning algorithms. However, their performance on precision is not satisfactory since they overestimate the possible CC tests and are insufficient in learning the deep semantic features. In this work, we propose a novel Triplet network-based Coincidental Correctness detection technique (i.e., TriCoCo) to overcome the limitations of the prior works. TriCoCo narrows the possible CC tests by designing three features to identify genuine passing tests. Instead of using all tests as inputs by existing techniques, TriCoCo takes the identified genuine passing tests and failing ones to train a triplet model that can evaluate their relative distance. Finally, TriCoCo infers the probability of being a CC test of the test in the rest of the passing tests by using the trained triplet model. We conduct large-scale experiments to evaluate TriCoCo based on the widely-used Defects4J benchmark. The results demonstrate that TriCoCo can improve not only the precision of CC detection but also the effectiveness of FL techniques, e.g., the precision of TriCoCo is 80.33% on average, and TriCoCo boosts the efficacy of DStar by 18%-74% in terms of MFR metric when compared to seven state-of-the-art CC detection baselines.