0% Complete
Home
/
14th International Conference on Computer and Knowledge Engineering
Distilled BERT Model In Natural Language Processing
Authors :
Yazdan Zandiye Vakili
1
Avisa Fallah
2
Hedieh Sajedi
3
1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
Keywords :
NLP،Machine Learning،Distillation،BERT،Transformers
Abstract :
This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.
Papers List
List of archived papers
Age Estimation Based on Facial Images Using Hybrid Features and Particle Swarm Optimization
NILOUFAR MEHRABI - SAYED PEDRAM HAERI BOROUJENI
PeQa: a Massive Persian Quenstion-Answering and Chatbot Dataset
Fatemeh Zahra Arshia - Mohammad Ali Keyvanrad - Saeedeh Sadat Sadidpour - Sayyid Mohammad Reza Mohammadi
FGM Copula based Analysis of Coverage Region for Wireless Three-User Multiple Access Channel with Correlated Channel Coefficients
Mona Sadat Mohsenzadeh - Ghosheh Abed Hodtani
Energy Efficient Power Allocation in MIMO-NOMA Systems with ZF Receiver Beamforming in Multiple Clusters
Mahdi Nangir - Abdolrasoul Sakhaei Gharagezlou - Nima Imani
Identification of Botnets and Nodes Attacking Smart Cities by Majority Voting Mechanism and Feature Selection
Maliheh Araghchi - Nazbanoo Farzaneh
Efficient Sub-Carrier Relationship Extraction for Human Activity Recognition via EEGNet in Wireless Sensing
Siavash Zaravashan - Sadegh ArefiZadeh - Sajjad Torabi
Analyzing the Impact of COVID-19 on Economy from the Perspective of User’s Reviews
Fatemeh Salmani - Hamed Vahdat-Nejad - Hamideh Hajiabadi
Damage Detection After the Earthquake Using Sentinel-1 and 2 Images and Machine Learning Algorithms (Case Study: Sarpol-e Zahab Earthquake)
Niloofar Alizadeh - Behnam Asghari Beirami - Mehdi Mokhtarzade
Forecasting El Niño Six Months in Advance Utilizing Augmented Convolutional Neural Network
Mohammad Naisipour - Iraj Saeedpanah - Arash Adib - Mohammad Hossein Neisi Pour
Improving the classification of high dimensional class-imbalanced data using the Chaos particle swarm optimization with Levy Flight
Mohammad Ali Zarif - Javad Hamidzadeh
more
Samin Hamayesh - Version 42.2.1