Validating large-scale lexical color resources

The use of the Web for crowd-sourcing lexical color resources has succeeded in creating databases consisting of millions of color terms. Various researchers have demonstrated the value of this data, but questions related to the quality and reliability of the data remain, because each large survey is tainted by a small number of disruptive subjects. The challenge is to cull the resource by identifying and eliminating the data contributed by these disruptive subjects. With a million color terms, it is no longer possible to individually inspect color terms and we need an automated process. Machine evaluation through natural language processing is possible, but this introduces the added complexity of pre-defining properties and criteria for data validity, which could improperly cloud the data. Color terms are terms associated with colors. Instead of examining the terms, we can examine their colors. Our visual system can process purely visual information at a much higher bandwidth, because the language system and its complex cognitive processes can be bypassed. In this contribution we propose a graphical approach in which the associated colors of large-scale lexical resources are first machine-sorted by color appearance so that human experts can efficiently identify outliers or questionable entries by simply looking at a graphical rendering. A recent test with the R. Munroe and E. Ellis Color Survey Data has allowed us to process over a million color terms. The methodology is as follows. First, the color terms are binned categorically, where each bin corresponds to a monolexemic color term. Second, for each term the associated red, green and blue sRGB values are further quantized and then these device values are sorted in lexicographical order. Third, the sorted device values are displayed as raster images in which each term is represented by a pixel drawn in the associated color. Finally, observers identify visually the outliers for each color term by inspecting the raster image. Using this procedure, the relatively rare disruptive subjects are efficiently identified and tagged. This process can be extended to multiple experts and a weight can be derived for the entries in the lexical resource. Experiments show that even using such crude appearance attributes as the sRGB values, the methodology is very effective and it is not crucial to use more sophisticated representations, such as for example correlates of hue, lightness and chroma. Based on this methodology, we show that the Munroe and Ellis Color Survey Data correlates well with data obtained in a controlled laboratory experiment. This is a surprising result given the informal nature of this resource. It is also a testimony of the validity of crowd-sourcing for scientific experimentation. © Copyright 2011 Hewlett-Packard Development Company, L.P.