0% Complete
Home
/
14th International Conference on Computer and Knowledge Engineering
Distilled BERT Model In Natural Language Processing
Authors :
Yazdan Zandiye Vakili
1
Avisa Fallah
2
Hedieh Sajedi
3
1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
Keywords :
NLP،Machine Learning،Distillation،BERT،Transformers
Abstract :
This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.
Papers List
List of archived papers
Adaptive Channel Estimation for MIMO-OFDM Systems in Impulsive Noise Environments
Mojtaba Hajiabadi
Low-Cost and Hardware Efficient Implementation of Pooling Layers for Stochastic CNN Accelerators
Mobin Vaziri - Hadi Jahanirad
A supervised approach using transformer networks for the detection of turning-related anomalies in urban intersections
Mohammad Mahdi HajiAbadi - Manoochehr Nahvi
A New Time Series Approach in Churn Prediction with Discriminatory Intervals
Hedieh Ahmadi - Seyed Mohammad Hossein Hasheminejad
A Federated Learning-Based Hybrid Deep Learning Framework for Enhanced Human Activity Recognition
Jamileh Azmoudeh - Sajjad Arghaee - Parisa Valizadeh - Samaneh Dandani - Iman Havangi - Mohammad Hossein Yaghmaee
Vaccine Distribution Modelling in Pandemics through Multi-Agent Systems: COVID-19 Case
Hossein Yarahmadi - Mohammad Ebrahim Shiri - Hamid Reza Navidi - Arash Sharifi - Moharram Challenger - Hassan Piriaei
Trust Management Enhancement for the Internet of Things: a Smart Contract Approach
Amin Rouzbahani - Fattaneh Taghiyareh
Recommending Popular Locations Based on Collected Trajectories
Mohammad Rabbani bidgoli - Saber Ziaei
Automatic Generation of XACML Code using Model-Driven Approach
Athareh Fatemian - Bahman Zamani - Marzieh Masoumi - Mehran Kamranpour - Behrouz Tork Ladani - Shekoufeh Kolahdouz Rahimi
A Language-Independent Approach to Classification of Textual File Fragments: Case Study of Persian, English, and Chinese Languages
Fatemeh Mansouri Hanis - Hamidreza Khoshvaghti - Mehdi Teimouri - Hadi Veisi
more
Samin Hamayesh - Version 41.3.1