Parametrization of an image understanding quality metric with a subjective evaluation

Image understanding has many real industrial applications (video-monitoring, image retrieval, etc.). Given an image and an associated ground truth, it is possible to quanjpgy the quality of understanding results provided by different algorithms or parameters. To this end, it is necessary to take into account many factors for each object in the image: localization and recognition errors and under or over-detection of objects. In order to define an evaluation metric for quanjpgying the quality of an image understanding result, we have to set, as for example, the weights of each kind of error in the global score. For a correct parameters setting of an evaluation metric we defined previously, we conducted a subjective evaluation of image understanding results involving many experts in image processing. We present in this paper the developed method and analyze the obtained results to weight the various errors in an appropriate way. We show the benefit of this kind of study to define the correct parameters of the metric in order to have a judgment as reliable the one provided by experts. Experimental results on many images from the PASCAL VOC Challenge show the good behavior of this metric compared to the human judgment. (C) 2012 Elsevier B.V. All rights reserved.