Paying participants: The impact of compensation on data quality

Abstract

Poor-quality data has the potential to increase error variance, reduce statistical power and effect sizes, and produce Type I or Type II errors. Paying participants is one technique researchers may use in an attempt to obtain high-quality data. Accordingly, two secondary datasets were used to examine the relationship between participant payment and data quality. The first dataset revealed that data quality did not differ between paid and unpaid undergraduates. Similarly, the second dataset showed that data quality did not differ between unpaid community participants and MTurkers. A comparison across stud-ies showed that undergraduate students engaged in lower levels of careless responding than the commu-nity samples but the unpaid community sample outperformed the MTurk sample and both undergraduate samples. Taken together, the current findings suggest that offering financial incentives to undergraduate or community samples does not improve data quality but may improve data collection rates and increase the diversity of participants.