Journal Article

Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool

Robinson, Jonathan, Rosenzweig, Cheskie, Litman, Leib, Moss, Aaron J

2019

Overview

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

• Publisher: Public Library of Science,Public Library of Science (PLoS)
• Subject: Adult / Behavioral Research - methods / Behavioral Research - standards / Bias / Biology and Life Sciences / Crowdsourcing - methods / Crowdsourcing - standards / Data Accuracy / Data collection / Data Collection - methods / Data Collection - standards / Datasets as Topic - standards / Female / Humans / Information management / Internet / Male / Medicine and Health Sciences / Methods / Middle Aged / Patient Selection / Physical Sciences / Practice Guidelines as Topic / Psychology / Quality / Research and Analysis Methods / Researchers / Retirement benefits / Sample Size / Sampling / Sampling Studies / Selection Bias / Social Sciences / Studies / Work / Workers / Young Adult