The Hidden Cost of Using Amazon Mechanical Turk for Research

Paper by Antonios Saravanos: “This work shares unexpected findings obtained from the use of the Amazon Mechanical Turk platform as a source of participants for the study of technology adoption. Expressly, of the 564 participants from the United States, 126 (22.34%) failed at least one of three forms of attention check (logic, honesty, and time). We also examined whether characteristics such as gender, age, education, and income affected participant attention. Amongst all characteristics assessed, only prior experience with the technology being studied was found to be related to attentiveness. We conclude this work by reaffirming the need for multiple forms of attention checks to gauge participant attention. Furthermore, we propose that researchers adjust their budgets accordingly to account for the possibility of having to discard responses from participants determined not to be displaying adequate attention….(More)”.