AI-powered content analysis: Using generative AI to measure media and communication content. Large language models (LLMs), starting with Google’s BERT, are increasingly being used to measure or classify media and communication content, particularly in the form of generative or conversational AI tools (e.g. OpenAI’s ChatGPT). The concept is straightforward yet fascinating: Rather than training and employing humans for annotation tasks, researchers describe the concept of interest to a model such as ChatGPT, present the coding unit and request a classification. The initial tests of ChatGPT's and similar tools' utility for content analysis yielded positive to enthusiastic results [1–3]. User-friendly tutorials have made the method accessible to the average social scientist [4, 5]. However, closed-source, commercial large language models are not fully understood, even by their developers, and their uncritical use has been criticised on ethical grounds [7–9]. In this seminar, we will engage with this cutting-edge research method in a practical way. We will start with a quick refresher on the basics of quantitative content analysis (both human and computational) and provide an overview of the rapidly developing literature on the utility of LLMs in this field. The majority of the seminar will focus on learning how to use and evaluate a generative AI model for applied content analysis research, step by step. By the end of the seminar, students should be able to apply the method to their own research.
References:
[1] Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120. https://doi.org/10/gsqx5m
[2] Heseltine, M., & Clemm von Hohenberg, B. (2024). Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1). https://doi.org/10/gtkhqr
[3] Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024). GPT is an effective tool for multilingual psychological text analysis. Proceedings of the National Academy of Sciences, 121(34), e2308950121. https://doi.org/10/gt7hrw
[4] Törnberg, P. (2024). Best practices for text annotation with large language models. Sociologica, 18(2), Article 2. https://doi.org/10/g9vgm7
[5] Stuhler, O., Ton, C. D., & Ollion, E. (2025). From codebooks to promptbooks: Extracting information from text with generative large language models. Sociological Methods & Research. https://doi.org/10/g9vgnq
[7] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? . Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10/gh677h
[8] Spirling, A. (2023). Why open-source generative AI models are an ethical way forward for science. Nature, 616(7957), 413–413. https://doi.org/10/gsqx6v
[9] Widder, D. G., Whittaker, M., & West, S. M. (2024). Why ‘open’ AI systems are actually closed, and why this matters. Nature, 635(8040), 827–833. https://doi.org/10/g8xdb3
