0% Complete
Home
/
14th International Conference on Computer and Knowledge Engineering
Distilled BERT Model In Natural Language Processing
Authors :
Yazdan Zandiye Vakili
1
Avisa Fallah
2
Hedieh Sajedi
3
1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
Keywords :
NLP،Machine Learning،Distillation،BERT،Transformers
Abstract :
This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.
Papers List
List of archived papers
A Graph-based Feature Selection using Class-Feature Association Map (CFAM)
Motahare Akhavan - Seyed Mohammad Hossein Hasheminejad
Optimizing the controller placement problem in SDN with uncertain parameters with robust optimization
Mohammad Kazemi - AhmadReza Montazerolghaem
Predicting the Recovery Rate of COVID-19 Using a Novel Hybrid Method
Fatemeh Ahouz - Ebrahim Sayahi
XAI for Transparent Autonomous Vehicles: A New Approach to Understanding Decision-Making in Self-driving Cars
Maryam Sadat Hosseini Azad - Amir Abbas Hamidi Imani - Shahriar Baradaran Shokouhi
Stock market prediction using multi-objective optimization
Mahshid Zolfaghari - Hamid Fadishei - Mohsen Tajgardan - Reza Khoshkangini
Capturing Local and Global Features in Medical Images by Using Ensemble CNN-Transformer
Javad Mirzapour Kaleybar - Hooman Saadat - Hooman Khaloo
Speech Emotion Recognition Using a Hierarchical Adaptive Weighted Multi-Layer Sparse Auto-Encoder Extreme Learning Machine with New Weighting and Spectral/SpectroTemporal Gabor Filter Bank Features
Fatemeh Daneshfar - Seyed Jahanshah Kabudian
Zone-Based Federated Learning in Indoor Positioning
Omid Tasbaz - Vahideh Moghtadaiee - Bahar Farahani
Ramp Progressive Secret Image Sharing using Ensemble of Simple Methods
Atieh Mokhtari - Mohammad Taheri
Non-Negative Matrix Factorization improves Residual Neural Networks
Hojjat Moayed
more
Samin Hamayesh - Version 41.5.3