Displacement Based Unsupervised Metric for Evaluating Rank Aggregation

Rank Aggregation is the problem of aggregating ranks given by various experts to a set of entities. In context of web, it has applications like building metasearch engines, combining user preferences etc. For many of these applications, it is difficult to get labeled data and the aggregation algorithms need to be evaluated against unsupervised evaluation metrics. We consider the Kendall-Tau unsupervised metric which is widely used for evaluating rank aggregation task. Kendall Tau distance between two permutations is defined as the number of pairwise inversions among the permutations. The original Kendall Tau distance treats each inversion equally, irrespective of the differences in rank positions of the inverted items. In this work, we propose a variant of Kendall-Tau distance that takes into consideration this difference in rank positions. We study, examine and compare various available supervised as well as unsupervised metrics with the proposed metric. We experimentally demonstrate that our modification in Kendall Tau Distance makes it potentially better than other available unsupervised metrics for evaluating aggregated ranking.