International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Distilled BERT Model In Natural Language Processing

Authors :

Yazdan Zandiye Vakili¹ Avisa Fallah² Hedieh Sajedi³

1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran

Keywords :

NLP،Machine Learning،Distillation،BERT،Transformers

Abstract :

This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.

Papers List

List of archived papers

ROCT-Net: A new ensemble deep convolutional model with improved spatial resolution learning for detecting common diseases from retinal OCT images

Mohammad Rahimzadeh - Mahmoud Reza Mohammadi

Underwater Image Super-Resolution using Generative Adversarial Network-based Model

Alireza Aghelan - Modjtaba Rouhani

Adaptive Channel Estimation for MIMO-OFDM Systems in Impulsive Noise Environments

Mojtaba Hajiabadi

Optimal PMU Placement Considering Reliability of Measurement System in Smart Grids

Mohammad Shahraeini - Shahla Khormali - Ahad Alvandi

EfficientNetB0’s Hybrid Approach for Brain Tumor Classification from MRI Images Using Deep Learning and Bagging Trees

Yeganeh Modaresnia - Farhad Abedinzadeh Torghabeh - Seyyed Abed Hosseini

Using Deep Learning for Classification of Lung Cancer on CT Images in Ardabil Province

Mohammad Ali Javadzadeh Barzaki - Jafar Abdollahi - Mohammad Negaresh - Maryam Salimi - Hadi Zolfeghari - Mohsen Mohammadi - Asma Salmani - Rona Jannati - Firouz Amani

MultiPath ViT OCR: A Lightweight Visual Transformer-based License Plate Optical Character Recognition

Alireza Azadbakht - Saeed Reza Kheradpisheh - Hadi Farahani

FarSick: A Persian Semantic Textual Similarity And Natural Language Inference Dataset

Zahra Ghasemi - Mohammad Ali Keyvanrad

Leveraging a structure-based and learning-based predictor using various feature groups in bioinformatics (case study: protein-peptide region residue-level interaction)

Shima Shafiee - Abdolhossein Fathi

An effective hybrid algorithm for locating splicing forgery image

Seyed Hesamoddin Hosseini - Amene Vatanparast - Amir Hossein Taherinia

Samin Hamayesh - Version 42.4.1