International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Distilled BERT Model In Natural Language Processing

Authors :

Yazdan Zandiye Vakili¹ Avisa Fallah² Hedieh Sajedi³

1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran

Keywords :

NLP،Machine Learning،Distillation،BERT،Transformers

Abstract :

This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.