Are Poor Quality Data Just Random Responses?: A Crowdsourced Study of Delay Discounting in Alcohol Use Disorder

Public Health Significance This study provides empirical evidence that poor quality delay discounting data does not differ from random responding in a sample of individuals with alcohol use disorder. This highlights that previous reports of poor quality data on Amazon Mechanical Turk extend to addiction-related samples. Thus, the use of rigorous data quality controls and exclusion of poor quality data are warranted to ensure high-quality scientific findings. Crowdsourced methods of data collection such as Amazon Mechanical Turk (MTurk) have been widely adopted in addiction science. Recent reports suggest an increase in poor quality data on MTurk, posing a challenge to the validity of findings. However, empirical investigations of data quality in addiction-related samples are lacking. In this study of individuals with alcohol use disorder (AUD), we compared poor quality delay discounting data to randomly generated data. A reanalysis of prior published delay discounting data was conducted comparing included, excluded, and randomly generated data samples. Nonsystematic criteria were implemented as a measure of data quality. The excluded data was statistically different from the included sample but did not differ from randomly generated data on multiple metrics. Moreover, a response bias was identified in the excluded data. This study provides empirical evidence that poor quality delay discounting data in an AUD sample is not statistically different from randomly generated data, suggesting data quality concerns on MTurk persist in addiction samples. These findings support the use of rigorous methods of a priori defined criteria to remove poor quality data post hoc. Additionally, it highlights that the use of nonsystematic delay discounting criteria to remove poor quality data is rigorous and not simply a way of removing data that does not conform to an expected theoretical model.