Benchmarking 360° Saliency Models by General-Purpose Metrics

How to effectively evaluate a model's capability to predict the visual attention of observers in 360 degrees scenes gains interest along with the advancement of saliency prediction modeling of omnidirectional images (ODIs). So far, many general-purpose metrics from 2D saliency literature have been adopted to evaluate the 360 degrees saliency models. However, whether they are still effective when being adopted to evaluate the 360 degrees saliency models has not been explored. In this paper, we testify several standard saliency evaluation metrics on the 360 degrees saliency models and comprehensively analyze their behaviors in the omnidirectional scenario. We find that 1) most metrics under-penalize false positives; 2) existing 360 degrees datasets involve severe equator bias that few metrics can effectively penalize. We hope this case study can provide a guideline for benchmarking 360 degrees image/video saliency models.