Please wait ...
0% Complete
Home
/
14th International Conference on Computer and Knowledge Engineering
Distilled BERT Model In Natural Language Processing
Authors :
Yazdan Zandiye Vakili
1
Avisa Fallah
2
Hedieh Sajedi
3
1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran
Keywords :
NLP،Machine Learning،Distillation،BERT،Transformers
Abstract :
This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.
Papers List
List of archived papers
Pruning and Mixed Precision Techniques for Accelerating Neural Network
Mahsa Zahedi - Mohammad Sediq Abazari Bozhgani - Abdorreza Savadi
Delay Optimization of a Federated Learning-based UAV-aided IoT network
Hossein Mohammadi Firouzjaei - Javad Zeraatkar Moghaddam - Mehrdad Ardebilipour
ROCT-Net: A new ensemble deep convolutional model with improved spatial resolution learning for detecting common diseases from retinal OCT images
Mohammad Rahimzadeh - Mahmoud Reza Mohammadi
MultiPath ViT OCR: A Lightweight Visual Transformer-based License Plate Optical Character Recognition
Alireza Azadbakht - Saeed Reza Kheradpisheh - Hadi Farahani
Mitochondrial Segmentation in Microscopy Images Using UNet-VGG19
Zerek Sediq Hossein - Rojiar Pir Mohammadiani - Saadat Izadi
Object Detection on Detecting Skin Lesion using Dab-DETR
Sheida Shadman - Amirreza Rouhbakhshmeghrazi - Shayan Nalbandian - Bo Li - Shaghayegh Shadman - Malik Muhammad Owais Siddique
A New Hypercube Variant: Pruned Shuffle Connected Cube
Reza Latifi - Mahmoud Naghibzadeh
CSI-Based Human Activity Recognition using Convolutional Neural Networks
Parisa Fard Moshiri - Mohammad Nabati - Reza Shahbazian - Seyed Ali Ghorashi
Innovative Customer Segmentation based on Multi-Step Sequential Deep Clustering in the Telecommunication Industry
Fatemeh Jalali Farahani - Shima Tabibian
An Efficient Planning Method for Autonomous Navigation of a Wheeled-Robot based on Deep Reinforcement Learning
Ali Salimi Sadr - Mahdi Shahbazi Khojasteh - Hamed Malek - Armin Salimi-Badr
more
Samin Hamayesh - Version 46.0.2