Combining conceptual query expansion and visual search results exploration for web image retrieval

Most approaches to image retrieval on the web have their basis in document search techniques. Images are indexed based on the text that is related to the images. Queries are matched to this text to produce a set of search results, which are organized in paged grids that are reminiscent of lists of documents. Due to ambiguity both with user-supplied queries and with the text used to describe the images within the search index, most image searches contain many irrelevant images distributed throughout the search results, and are often focused on the most common interpretation of the query. We propose a method for addressing these problems in which conceptual query expansion is used to generate a diverse range of images, and a multi-resolution extension of a self-organizing map is used to group visually similar images. The resulting interface acts as an intelligent search assistant, automatically diversifying the search results and then allowing the searcher to interactively highlight and filter images based on the concepts, and zoom into an area within the image space to show additional images that are visually similar. Evaluations show that the precision of the image search results increase as a result of concept-based focusing and filtering, as well as visual zooming operations, even for uncommon interpretations of ambiguous queries.