Testing ‘Crowdcoding’ Methods in Sub-Saharan African Settings: Using the 2020 Tanzanian Elections to Test its Validity and Reliability

Testing ‘Crowdcoding’ Methods in Sub-Saharan African Settings:
Using the 2020 Tanzanian Elections to Test its Validity and Reliability

Gregory Gondwe & Evariste Some
University of Colorado Boulder, USA

Abstract
This study replicates existing research on crowdcoding, and content analysis approaches to test the validity and reliability of content analysis methods in the African setting. We use data from the 2020 Tanzanian presidential elections as a case study. Instead of MTurk for crowdsourcing, the study utilized WhatsApp groups and university students from Tanzania to code the data. Using a collected and controlled sample of 400 tweets to represent Tanzania’s ruling and opposition parties, respectively, our overall findings suggested that crowdcoding produced more reliable data than qualitative content analysis (QCA). However, further analysis suggests that although Crowdcoding recorded higher agreement on validity scores, trained coders seemed to provide more reliability accuracy scores. Besides, data indicates that the traditional training of the coders was statistically insignificant in providing accurate validity and reliability scores for QCA.

Keywords:Crowdcoding, crowdsourcing, sentiment analysis, content analysis, Tanzania, sub-Saharan Africa

References

Alikhani, E. (2014). Computational Social Analysis: Social Unrest Prediction Using Textual Analysis of News. State University of New York at Binghamton

Agarwal, S., & Sureka, A. (2016, August). But i did not mean it!—intent classification of racist posts on tumblr. In 2016 European Intelligence and Security Informatics Conference (EISIC) (pp. 124-127). IEEE.

Du Plessis, C. (2011). A computational text analysis of the South African banking sector’s representation of its core values: A corpus-driven approach. Communication, 37(3), 422-442.

Golan, G., & Wanta, W. (2001). Second-level agenda setting in the New Hampshire primary: A comparison of coverage in three newspapers and public perceptions of candidates. Journalism & Mass Communication Quarterly, 78(2), 247-259.

Gondwe, G., Muchangwe, R., & Mwaya, J. E. Motivations for Social Media Use and Consumption in Zambian Online Platforms. In Analyzing Global Social Media Consumption (pp. 204-215). IGI Global.

Gondwe, G. (2020). Agenda-Setting theory in African contexts: A Jekyll and Hyde in the Zambian Presidential elections. International Journal of Multidisciplinary Research and Development, 7(5), 93-100.

Gondwe, G. (2018). News Believability & Trustworthiness on African Online Networks: An Experimental Design. International Communication Research Journal, 53(2), 51-74

Guo, L., Mays, K., Lai, S., Jalal, M., Ishwar, P., & Betke, M. (2020). Accurate, Fast, But Not Always Cheap: Evaluating “Crowdcoding” as an Alternative Approach to Analyze Social Media Data. Journalism & Mass Communication Quarterly, 97(3), 811–834. https://doi.org/10.1177/1077699019891437

Guo, L., Mays, K., Sameki, M., & Betke, M. (2017). From crowdsourcing to crowdcoding: An alternative approach to annotate big data in communication research. Paper presented at the ICA annual conference, San Diego, May 2017.

Kerunga, J., Rowe, E., & Gondwe, G. (2020). Journalism Credibility in the digital age–Examining shifts in paradigms. Available at SSRN 3641943.

Krippendorff, K. (2013). Content analysis. An introduction to its methodology. Thousand Oaks, CA: Sage. Levay, K. E., Freese, J., & Druckman, J. N. (2016). The demographic and political composition of mechanical turk samples. SAGE Open, 6, 1–17. doi:10.1177/2158244016636433

Kwansah-Aidoo, K. (2001). The appeal of qualitative methods to traditional agenda-setting research: An example from West Africa. Gazette (Leiden, Netherlands), 63(6), 521-537.

Lacy, S., Watson, B. R., Riffe, D., & Lovejoy, J. (2015). Issues and best practices in content analysis. Journalism & Mass Communication Quarterly, 92(4), 791–811.

Lind, F., Gruber, M., & Boomgaarden, H. G. (2017). Content analysis by the crowd: Assessing the usability of crowdsourcing for coding latent constructs. Communication methods and measures, 11(3), 191-209.

McCombs, M. E., & Shaw, D. L. (1972). The agenda-setting function of mass media. Public opinion quarterly, 36(2), 176-187.

Madrid-Morales, D. (2020). Using Computational Text Analysis Tools to Study African Online News Content. African Journalism Studies, 1-15

Neuendorf, K. A. (2017). The content analysis guidebook. SAGE

Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage. Oosterman, J., Yang, J., Bozzon, A., Aroyo, L., & Houben, G. J. (2015). On the impact of knowledge extraction and aggregation on crowd sourced annotation of visual artworks. Computer Networks, 90, 133–149. doi:10.1016/j.comnet.2015.07.008

Riffe, D., Lacy, S., & Fico, F. (2014). Analyzing media messages: Using quantitative content analysis in research. Routledge.

Sikanku, G. E. (2014). Consolidating inter-media agenda-setting research in Ghana: A study of associational relationships among wire, online, and print news media. Journal of Black Studies, 45(5), 396-414.

Some, E., Gondwe, G., & Rowe, E. W. (2019, October). Cybersecurity and Driverless Cars: In Search for a Normative Way of Safety. In 2019 Sixth International Conference on Internet of Things: Systems, Management and Security (IOTSMS) (pp. 352-357). IEEE.

Gregory Gondwe is an affiliate of the Department of Journalism at the University of Colorado-Boulder, USA where he teaches media-related courses. His research explores the concepts of persuasive messages and their effects on the news. He examines how individuals and groups choose to assimilate and accept news content as either true or false with a particular interest in Sub-Saharan Africa. His current research explores the Chinese news agenda in African media systems.

Evariste Some is a Ph.D. candidate of Computer Science at the University of Colorado – Boulder, USA. His current research focus is on multiple-input multiple-output or massive MIMO and beamforming.

Correspondence to: Gregory Gondwe, College of Media, Communication, and Information (CMCI), Department of Journalism, University of Colorado Boulder, 1511 University Avenue, USA.