Comparing approaches to analyze refactoring activity on software repositories

Some approaches have been used to investigate evidence on how developers refactor their code, whether refactorings activities may decrease the number of bugs, or improve developers' productivity. However, there are some contradicting evidence in previous studies. For instance, some investigations found evidence that if the number of refactoring changes increases in the preceding time period the number of defects decreases, different from other studies. They have used different approaches to evaluate refactoring activities. Some of them identify committed behavior-preserving transformations in software repositories by using manual analysis, commit messages, or dynamic analysis. Others focus on identifying which refactorings are applied between two programs by using manual inspection or static analysis. In this work, we compare three different approaches based on manual analysis, commit message (Ratzinger's approach) and dynamic analysis (SAFEREFACTOR'S approach) to detect whether a pair of versions determines a refactoring, in terms of behavioral preservation. Additionally, we compare two approaches (manual analysis and REF-FINDER) to identify which refactorings are performed in each pair of versions. We perform both comparisons by evaluating their accuracy, precision, and recall in a randomly selected sample of 40 pairs of versions of JHotDraw, and 20 pairs of versions of Apache Common Collections. While the manual analysis presents the best results in both comparisons, it is not as scalable as the automated approaches. Ratzinger's approach is simple and fast, but presents a low recall; differently, SAFEREFACTOR is able to detect most applied refactorings, although limitations in its test generation backend results for some kinds of subjects in low precision values. REF-FINDER presented a low precision and recall in our evaluation. (c) 2012 Elsevier Inc. All rights reserved.