International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Distilled BERT Model In Natural Language Processing

Authors :

Yazdan Zandiye Vakili¹ Avisa Fallah² Hedieh Sajedi³

1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran

Keywords :

NLP،Machine Learning،Distillation،BERT،Transformers

Abstract :

This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.

Papers List

List of archived papers

TriFuse-PdM: High-Fidelity Machine Failure Prediction Using Hybrid Resampling and Model Calibration

Saghar Shafaati - Javad Mohammadzadeh

An Automated Visual Defect Segmentation for Flat Steel Surface Using Deep Neural Networks

Dorna Nourbakhsh Sabet - Mohammad Reza Zarifi - Javad Khoramdel - Yasamin Borhani - Esmaeil Najafi

Taguchi Design of Experiments Application in Robust sEMG Based Force Estimation

Mohsen Ghanaei - Hadi Kalani - Alireza Akbarzadeh

Explainable Error Detection Method for Structured Data using HoloDetect framework

Abolfazl Mohajeri Khorasani - Sahar Ghassabi - Behshid Behkamal - Mostafa Milani

A Novel Density-Based KNN in Pattern Recognition

Sajad Haghzad Klidbary - Abazar Arabameri

Object Detection on Detecting Skin Lesion using Dab-DETR

Sheida Shadman - Amirreza Rouhbakhshmeghrazi - Shayan Nalbandian - Bo Li - Shaghayegh Shadman - Malik Muhammad Owais Siddique

Cardiology Disease Diagnosis by Analyzing Histological Microscopic Images Using Deep Learning

Maria Salehpanah - Jafar Tanha - Zahra Jafari - SeyedEhsan Roshan - Sajad Rezaei

SUT: a new multi-purpose synthetic dataset for Farsi document image analysis

Elham Shabaninia - Fatemeh sadat Eslami - Ali Afkari Fahandari - Hossein Nezamabadi-pour

Enhanced Autoencoder-based Clustering for Message Analysis in Binary Protocols

Mohaddese Nemati - Shiva Mahmoudzadeh - Mehdi Teimouri

Designing an IT2 Fuzzy Rule-based System for Emotion Recognition Using Biological Data

Mahsa Keshtkar - Hooman Tahayori