Journal Article

Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillation

Karim, Ahmed Akib Jawad, Asad, Kazi Hafiz Md, Alam, Md. Golam Rabiul

2025

Overview

This work focuses on the efficiency of the knowledge distillation approach in generating a lightweight yet powerful BERT-based model for natural language processing (NLP) applications. After the model creation, we applied the resulting model, LastBERT, to a real-world task—classifying severity levels of Attention Deficit Hyperactivity Disorder (ADHD)-related concerns from social media text data. Referring to LastBERT, a customized student BERT model, we significantly lowered model parameters from 110 million BERT base to 29 million-resulting in a model approximately 73.64% smaller. On the General Language Understanding Evaluation (GLUE) benchmark, comprising paraphrase identification, sentiment analysis, and text classification, the student model maintained strong performance across many tasks despite this reduction. The model was also used on a real-world ADHD dataset with an accuracy of 85%, F1 score of 85%, precision of 85%, and recall of 85%. When compared to DistilBERT (66 million parameters) and ClinicalBERT (110 million parameters), LastBERT demonstrated comparable performance, with DistilBERT slightly outperforming it at 87%, and ClinicalBERT achieving 86% across the same metrics. These findings highlight the LastBERT model’s capacity to classify degrees of ADHD severity properly, so it offers a useful tool for mental health professionals to assess and comprehend material produced by users on social networking platforms. The study emphasizes the possibilities of knowledge distillation to produce effective models fit for use in resource-limited conditions, hence advancing NLP and mental health diagnosis. Furthermore underlined by the considerable decrease in model size without appreciable performance loss is the lower computational resources needed for training and deployment, hence facilitating greater applicability. Especially using readily available computational tools like Google Colab and Kaggle Notebooks. This study shows the accessibility and usefulness of advanced NLP methods in pragmatic world applications.

• Publisher: Public Library of Science,Public Library of Science (PLoS)
• Subject: Accuracy / Attention Deficit Disorder with Hyperactivity - classification / Attention Deficit Disorder with Hyperactivity - diagnosis / Attention Deficit Disorder with Hyperactivity - psychology / Attention deficit hyperactivity disorder / Biology and Life Sciences / Classification / Cognitive ability / Complications and side effects / Computational linguistics / Computer and Information Sciences / Computer applications / Datasets / Deep learning / Distillation / Efficiency / Electronic health records / Humans / Knowledge / Knowledge acquisition (Expert systems) / Language / Language processing / Machine learning / Medical personnel / Medical research / Medicine and Health Sciences / Medicine, Experimental / Mental disorders / Mental health / Methods / Natural language interfaces / Natural Language Processing / Neural networks / Parameter identification / People and Places / Physiological aspects / Research and Analysis Methods / Sentiment analysis / Severity of Illness Index / Social Media / Social networks / Social organization / Social Sciences / Software / Technology application / Text categorization