Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics
Keywords:Crowdworkers, Micro Tasks, Demographics, Cross-National Comparison
AbstractSince its emergence roughly a decade ago, microtask crowdsourcing has been attracting a heterogeneous set of workers from all over the globe. This paper sets out to explore the characteristics of the international crowd workforce and offers a cross-national comparison of crowdworker populations from ten countries. We provide an analysis and comparison of demographic characteristics and shed light on the significance of microtask income for workers situated in different national contexts. With over 11,000 individual responses, this study is the first large-scale country-level analysis of the characteristics of workers on the platform Appen (formerly CrowdFlower and Figure Eight), one of the two platforms dominating the microtask market. We find large differences between the characteristics of the crowd workforces of different countries, both regarding demography and regarding the importance of microtask income for workers. Furthermore, we find that the composition of the workforce in the ten countries was largely stable across samples taken at different points in time.
Amazon Mechanical Turk (2016). Amazon Mechanical Turk: Worker Web Site FAQs. http://www.mturk.com/mturk/help?helpPage=
worker#how_paid. (2016). Accessed: 2016-12-20.
Berg, J. (2016). Income security in the on-demand economy: findings and policy lessons from a survey of crowdworkers. (2016).
Berinsky, A. J, Huber, G. A, and Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com’s
Mechanical Turk. Political Analysis 20, 3 (2012), 351–368.
Bloomberg News (2016). Venezuela’s Currency Is Collapsing on the Black Market Again. http://www.bloomberg.com/news/articles/
-11-01/venezuela-s-currency-is-collapsing-on-the-black-market-again. (2016). Accessed: 2016-12-20.
Brabham, D. C. (2010). MOVING THE CROWDAT THREADLESS. Information, Communication & Society 13, 8 (2010), 1122–1145.
Brewer, R, Morris, M. R, and Piper, A. M. (2016). Why would anybody do this?: Understanding older adults’ motivations and challenges in crowd work. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2246–2257.
Buhrmester, M, Kwang, T, and Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on psychological science 6, 1 (2011), 3–5.
Destatis. (2013). Einkommens- und Verbrauchsstichprobe: Einnahmen und Ausgaben privater Haushalte. Fachserie 15 des Statistischen Bundesamtes 4 (2013).
Difallah, D, Filatova, E, and Ipeirotis, P. (2018). Demographics and dynamics of mechanical turk workers. In Proceedings of the
Eleventh ACM International Conference on Web Search and Data Mining. ACM, 135–143.
European Agency for Health and Safety at Work. (2015). The future of work: crowdsourcing. https://osha.europa.eu/en/
tools-and-publications/publications/future-work-crowdsourcing/view. (2015). Accessed: 2018-10-20.
Goodman, J. K and Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research 44, 1 (2017), 196–210.
Hirth, M, Hoßfeld, T, and Tran-Gia, P. (2011). Human cloud as emerging Internet application-anatomy of the microworkers crowdsourcing platform. University of Wurzburg Institute of Computer Science Research Report Series (2011).
Huff, C and Tingley, D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics 2, 3 (2015), 2053168015604648.
Ipeirotis, P. G. (2010)a. Analyzing the Amazon Mechanical Turk marketplace. ACM Crossroads 17, 2 (2010), 16–21. DOI:http:
Ipeirotis, P. G. (2010)b. Demographics of mechanical turk. CeDER Working Papers (2010).
Kazai, G, Kamps, J, and Milic-Frayling, N. (2012). The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy. In Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 2583–2586.
Kuek, S. C, Paradi-Guilford, C, Fayomi, T, Imaizumi, S, Ipeirotis, P, Pina, P, Singh, M, and others, . (2015). The global opportunity in online outsourcing. Technical Report. The World Bank.
Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145–151.
Martin, D, Carpendale, S, Gupta, N, Hoßfeld, T, Naderi, B, Redi, J, Siahaan, E, and Wechsung, I. (2017). Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing. In Evaluation in the Crowd. Crowdsourcing and Human- Centered Experiments. Springer, 27–69.
Naderi, B. (2018). Who are the Crowdworkers? In Motivation of Workers on Microtask Crowdsourcing Platforms. Springer, 17–27.
Paolacci, G and Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science 23, 3 (2014), 184–188.
Paolacci, G, Chandler, J, and Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk. Judgment and Decision making 5, 5 (2010), 411–419.
Pavlick, E, Post, M, Irvine, A, Kachaev, D, and Callison-Burch, C. (2014). The language demographics of amazon mechanical turk.
Transactions of the Association for Computational Linguistics 2 (2014), 79–92.
Peer, E, Brandimarte, L, Samat, S, and Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral
research. Journal of Experimental Social Psychology 70 (2017), 153–163.
Posch, L, Bleier, A, and Strohmaier, M. (2017). Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Moti-vation Scale. CoRRhttps://arxiv.org/abs/1702.01661 (2017).
Ross, J, Irani, L, Silberman, M. S, Zaldivar, A, and Tomlinson, B. (2010). Who are the crowdworkers?: shifting demographics in mechanical turk. In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume, Atlanta, Georgia, USA, April 10-15, 2010, Elizabeth D. Mynatt, Don Schoner, Geraldine Fitzpatrick, Scott E. Hudson, W. Keith Edwards, and Tom Rodden (Eds.). ACM, 2863–2872. DOI:http://dx.doi.org/10.1145/1753846.1753873
Ross, J, Zaldivar, A, Irani, L, and Tomlinson, B. (2009). Who are the turkers? worker demographics in amazon mechanical turk.
Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009).
Shapiro, D. N, Chandler, J, and Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science 1, 2 (2013), 213–220.
United Nations Department Of Economic And Social Affairs - Statistics Division. (2000). Classifications of expenditure according to purpose. https://unstats.un.org/unsd/publication/SeriesM/SeriesM_84E.pdf. (2000). Accessed: 2018-12-02.
U.S. Bureau of Labor Statistics (2018). Consumer Expenditure Surveys - CE Survey Materials. https://www.bls.gov/cex/
csxsurveyforms.htm. (2018). Accessed: 2018-12-02.
Weinberg, J. D, Freese, J, and McElhattan, D. (2014). Comparing Data Characteristics and Results of an Online Factorial Survey
between a Population-based and a Crowdsource-recruited Sample. Sociological Science 1 (2014).
How to Cite
Copyright (c) 2022 Human Computation
This work is licensed under a Creative Commons Attribution 4.0 International License.Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).