0% Complete
Home
/
13th International Conference on Computer and Knowledge Engineering
Enhancing Lighter Neural Network Performance with Layer-wise Knowledge Distillation and Selective Pixel Attention
Authors :
Siavash Zaravashan
1
Sajjad Torabi
2
Hesam Zaravashan
3
1- Iran University of Science and Technology
2- Part AI Research Center
3- Iran University of Science and Technology
Keywords :
Knowledge distillation،Image classification،Deep learning،Model compression،Computer vision
Abstract :
Deep neural networks have made a revolution in different areas of Artificial Intelligence such as image classification and object detection. However, most of these models are heavy and time-consuming at the inference stage. If we desire to use real-time applications, it's necessary to lighten them up. However, this could negatively affect the performance of models. One suggested approach for enhancing the capabilities of smaller models involves a technique called knowledge distillation. In this method, a less advanced model, referred to as the student model, learns from and emulates the more advanced teacher model to enhance its performance. This method has been proven as an effective technique to compress the model and increase its accuracy. Previous methods mostly focus on proposing feature transformation and loss functions between the same level’s features to improve the effectiveness. In this paper, we investigate the effect of connections between different layers of teacher and student networks and reveal their great importance. We have employed three different loss functions—attention-driven distillation, hierarchical context loss, and non-local loss—to enhance the classifier's overall performance. Extensive experiments on the CIFAR-100 dataset show that the proposed method has a great performance improvement. For the benefit of the research community, we will make the code for this study available on GitHub.
Papers List
List of archived papers
DIPT: Diversified Personalized Transformer for QAC systems
Mahdi Dehghani - Samira Vaez Barenji - Saeed Farzi
An Ensemble CNN for Brain Age Estimation based on Hippocampal Region Applicable to Alzheimer's Diagnosis
Zahra Qodrati - Seyedeh Masoumeh Taji - Habibollah Danyali - Kamran Kazemi
Traffic Sign Recognition Using Local Vision Transformer
Ali Farzipour - Omid Nejati Manzari - Shahriar B. Shokouhi
Optimizing MR Image Registration for Accurate Brain Volume Measurement in Children with Autism Spectrum Disorder
Shiva Sanati - Mahdi Saadatmand
BERT transformers Multitask learning Sarcasm and Sentiment classification (BMSS)
Fatemeh Molavi - Jamshid Bagherzadeh Mohasefi
Emotion Recognition In Persian Speech Using Deep Neural Networks
Ali Yazdani - Hossein Simchi - Yasser Shekofteh
A Framework for Automated Cardiovascular Magnetic Resonance Image Quality Scoring based on EuroCMR Registry Criteria
Shahabedin Nabavi - Mohsen Ebrahimi Moghaddam - Ahmad Ali Abin - Alejandro Frangi
Extracting Major Topics of COVID-19 Related Tweets
Faezeh Azizi - Hamed Vahdat-Nejad - Hamideh Hajiabadi - Mohammad Hossein Khosravi
Sensitivity Reliability Analysis of Power Distribution Networks Using Fuzzy Logic
Mohammed Wadi - Wisam Elmasry - Ismail Kucuk - Hossein Shahinzadeh
An Analysis of Botnet Detection Using Graph Neural Network
Faezeh Alizadeh - Mohammad Khansari
more
Samin Hamayesh - Version 41.3.1