Framing Effects and Preference Reversals in Crowd-Sourced Ranked Opinions

Framing effects influence the measurement of knowledge, preference, and opinions, and they have implications for understanding mental representation and cognitive processes. Much of the evidence for framing effects, however, comes from controlled laboratory experiments rather than naturally occurring real-world behavior. We demonstrate how a crowd-sourced opinion website provides naturally occurring data to test for preference reversals and framing effects in subjective preferences. The data take the form of people's top-n rankings of lists of items in response to specific questions, and we consider related pairs of lists for which the same people provide rankings. We compare the list "best actors in film history" with "best actors working today" and the list "best National Basketball Association (NBA) players of all time" with "best white NBA players of all time." These lists have a conjunctive relationship, since the second list asks the same question as the first, but about a restricted set of items. We also compare the list "best U.S. presidents of all time" with "worst U.S. presidents." These lists ask the same question but use "best" versus "worst" contexts. For all three comparisons, we find many cases in which the same person reverses their preferences for pairs of items between lists, and for which the same set of people systematically change their rate of ordering pairs of items. Our results provide real-world evidence for the presence of preference reversals and framing effects in subjective preferences. They also provide an example of the strengths and limitations of studying choice and preference using naturally occurring data.