Journal Article

DPAL-BERT: A Faster and Lighter Question Answering Model

AlQahtani, Salman A., AlSanad, Ahmed, Yin, Zhengtong, Cai, Zhuohang, Li, Xiaolu, Yin, Lirong, Chen, Xiaobing, Wang, Ruiyang, Wang, Lei, Zheng, Wenfeng, Lu, Siyu

2024

Overview

Recent advancements in natural language processing have given rise to numerous pre-training language models in question-answering systems. However, with the constant evolution of algorithms, data, and computing power, the increasing size and complexity of these models have led to increased training costs and reduced efficiency. This study aims to minimize the inference time of such models while maintaining computational performance. It also proposes a novel Distillation model for PAL-BERT (DPAL-BERT), specifically, employs knowledge distillation, using the PAL-BERT model as the teacher model to train two student models: DPAL-BERT-Bi and DPAL-BERT-C. This research enhances the dataset through techniques such as masking, replacement, and n-gram sampling to optimize knowledge transfer. The experimental results showed that the distilled models greatly outperform models trained from scratch. In addition, although the distilled models exhibit a slight decrease in performance compared to PAL-BERT, they significantly reduce inference time to just 0.25% of the original. This demonstrates the effectiveness of the proposed approach in balancing model performance and efficiency.

• Publisher: Tech Science Press
• Subject: Accuracy / Datasets / Distillation / Efficiency / Evolutionary algorithms / Inference / Knowledge / Language / Natural language processing / Probability distribution / Questions