A new method for automatic performance comparison of search engines

In this paper, we present a new method for automatically comparing the performance, such as precision, of search engines. Based on queries randomly selected from a specific domain of interest, the method uses robots to automatically query the target search engines, evaluates the relevance of the returned links to the query either automatically based on the vector space model or manually, and then applies statistic measures, including the probability of win and the Friedman statistic, to compare the performance of search engines. We show the experimental results of the new method on three search engines, AltaVista, Google, and InfoSeek. The method arrived at the same performance comparison result in applying either the automatic relevance evaluation method or the manual method. In addition, our results show that the probability of win is a better metric than the Friedman statistic in performance comparison. The advantage of the new method is that it is fast, flexible, consistent, and can adapt to the fast changing search engines. © 2000, Kluwer Academic Publishers.