0% Complete
Home
/
14th International Conference on Computer and Knowledge Engineering
Distilled BERT Model In Natural Language Processing
Authors :
Yazdan Zandiye Vakili
1
Avisa Fallah
2
Hedieh Sajedi
3
1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
Keywords :
NLP،Machine Learning،Distillation،BERT،Transformers
Abstract :
This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.
Papers List
List of archived papers
Identification of Botnets and Nodes Attacking Smart Cities by Majority Voting Mechanism and Feature Selection
Maliheh Araghchi - Nazbanoo Farzaneh
Deep Inside Tor: Exploring Website Fingerprinting Attacks on Tor Traffic in Realistic Settings
Amirhossein Khajehpour - Farid Zandi - Navid Malekghaini - Mahdi Hemmatyar - Naeimeh Omidvar - Mahdi Jafari Siavoshani
A Hybrid Echo State Network for Hypercomplex Pattern Recognition, Classification, and Big Data Analysis
Mohammad Jamshidi - Fatemeh Daneshfar
Machine and Deep Learning Models for Prediction of Small Molecule–Biotech Drug Pair’s Interactions
Fatemeh Nasiri - Mohsen Hooshmand
Experimental evaluation and comparison of anti-pattern detection tools by the gold standard
Somayeh Kalhor - Mohammad reza Keyvanpour - Afshin Salajegheh
Real-Time Forecasting Using Mixed Frequency Time-Series Data
Armin Khayati - Mohammad Taheri - Koorush Ziarati
Adversarial Robustness Evaluation with Separation Index
Bahareh Kaviani Baghbaderani - Afsaneh Hasanebrahimi - Ahmad Kalhor - Reshad Hosseini
A Self-Configurable Model for Cloud Resource Allocation
Ali Bazghandi
Analyzing the Impact of COVID-19 on Economy from the Perspective of User’s Reviews
Fatemeh Salmani - Hamed Vahdat-Nejad - Hamideh Hajiabadi
New Design of Efficient Reversible Quantum Saturation Adder
Negin Mashayekhi - Mohammad Reza Reshadinezhad - Shekoofeh Moghimi
more
Samin Hamayesh - Version 42.2.1