International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Distilled BERT Model In Natural Language Processing

Authors :

Yazdan Zandiye Vakili¹ Avisa Fallah² Hedieh Sajedi³

1- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 2- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran 3- School of Mathematics, Statistics and Computer Science, University of Tehran, Tehran, Iran

Keywords :

NLP،Machine Learning،Distillation،BERT،Transformers

Abstract :

This paper reviews the evolution of Natural Language Processing (NLP) models, focusing on the distillation techniques used to create efficient and compact versions of large models. Traditional NLP models laid the foundation but had limitations in scalability and contextual understanding. Transformer models like BERT revolutionized NLP but required significant computational resources. This review examines TinyBERT, DistilBERT, MobileBERT, and MiniLM, which balance size and performance through knowledge distillation. These distilled models maintain high performance while being suitable for deployment on resource-constrained devices, making advanced NLP capabilities accessible in real-world applications.

Papers List

List of archived papers

Community-Based QoE Enhancement for User-Generated Content Live Streaming

Reza Saeedinia - S.Omid Fatemi - Daniele Lorenzi - Farzad Tashtarian - Christian Timmerer

Decentralized Federated Learning in IoT Environments: A Hierarchical Approach

Majid Mohammadpour - Seyedakbar Mostafavi

A Hybrid Architecture to Optimize Persian FAQ Retrieval using Semantic Similarity Search

Seyed Amir Mohammad Hosseini - Fatemeh Dehbashi - Setare Kahnemuee - Mohsen Kahani - Morteza Fardin

Damage Detection After the Earthquake Using Sentinel-1 and 2 Images and Machine Learning Algorithms (Case Study: Sarpol-e Zahab Earthquake)

Niloofar Alizadeh - Behnam Asghari Beirami - Mehdi Mokhtarzade

LightFedSelect: A Lightweight Framework for Byzantine-Robust Federated Learning

Seyed Saeed Razavi - Seyed Arsalan Vasegh Rahim Parvar - Soroosh Dadashi Pakdeh - Mohammad Matin Rezaeifard - Morteza Mollaie Chafi - Reza Ebrahimi Atani

A New Time Series Approach in Churn Prediction with Discriminatory Intervals

Hedieh Ahmadi - Seyed Mohammad Hossein Hasheminejad

An Efficient Approach for Breast Abnormality Detection through High-Level Features of Thermography Images

Farhad Abedinzadeh Torghabeh - Yeganeh Modaresnia - Seyyed Abed Hosseini

Variance-Guided Feature Correlation for Deep Full-Reference Image Quality Assessment

Amirreza Khakpour - Sina Yademellat - Azadeh Mansouri

SingAll: Scalable Control Flow Checking for Multi-Process Embedded Systems

Mehdi Amininasab - Ahmad Patooghy - Mahdi Fazeli

Low-Cost and Hardware Efficient Implementation of Pooling Layers for Stochastic CNN Accelerators

Mobin Vaziri - Hadi Jahanirad