Journal Article

LLM-Based Classification and Topic Modeling of Generative AI Ethical Risk Discourse on Social Media

Kim, Soyon, Lee, Bong Gyou, Kim, Soo Hyung, Yoon, Cheolhee

2026

Overview

Identifying sparse, context-dependent target discourse such as ethical risk discourse on generative AI in noisy social media data is a long-standing challenge in computational text analysis. Beyond classification, interpreting the resulting topic structure transparently poses an additional methodological challenge. To address these problems, this study proposes a reproducible two-stage framework. First, four LLMs (GPT-4.1, GPT-3.5-turbo, Claude Sonnet 4.6, and Gemini 2.5 Pro) are compared on a human-annotated validation set using a zero-shot prompt grounded in five ethical risk categories, and the best-performing model is selected to classify the full keyword-prescreened corpus. Validity is evaluated against human annotations and supervised baselines. Second, BERTopic is applied to the classified corpus, and sub-topics are linked to higher-level categories through an explicit mapping protocol. The study draws on approximately 500,000 ChatGPT-related English-language tweets from January to March 2023. Only 36.4% of the validation sample constituted ethical risk discourse, confirming substantial pre-screening noise. GPT-4.1, selected as the final classification model, achieved the strongest performance (accuracy = 0.899, F1 = 0.859, Cohen’s κ = 0.780), significantly outperforming conventional supervised baselines. BERTopic yielded 33 non-outlier sub-topics; within this subset, Societal and democratic risks (36.16%) and Technical safety (20.90%) were the largest categories. These findings should be interpreted not as a direct measure of users’ risk perceptions in 2023, but as the category distribution within the structured subset excluding outliers, after keyword screening and classification under the current ethical risk taxonomy used in this study. Within this scope, societal and institutional concerns were at least as prominent as those related to technical failures.