International Conference on Computer and Knowledge Engineering

Home / 13th International Conference on Computer and Knowledge Engineering

Facial Emotion Recognition Under Mask Coverage Using a Data Augmentation Technique

Authors :

Aref Farhadipour¹ Pouya Taghipour²

1- IRI Broadcasting University 2- École de technologie supérieure (ETS) University

Keywords :

Facial Emotion Recognition،Convolutional Neural Network،Data Augmentation،Transfer Learning

Abstract :

Identifying human emotions using AI-based computer vision systems, when individuals wear face masks, presents a new challenge in the current Covid-19 pandemic. In this study, we propose a facial emotion recognition system capable of recognizing emotions from individuals wearing different types of face masks. A novel data augmentation technique was utilized to improve the performance of our model using four mask types for each face image. We evaluated the effectiveness of three convolutional neural networks, namely Alexnet, Squeezenet, and VGGFace2, that were trained using transfer learning. The experimental findings revealed that our model works effectively in multi-mask mode compared to single-mask mode. The VGGFace2 network achieved the highest accuracy rate, with 97.82% for the person-dependent mode and 74.21% for the person-independent mode using the JAFFE dataset. Additionally, we evaluated our proposed model using the UIBVFED dataset. Once again, the VGGFace2 network has demonstrated superior performance, with accuracies of 71.57% for the person-dependent mode and 57.44% for the person-independent mode. Moreover, we employed metrics such as precision, sensitivity, specificity, AUC, F1 score, and confusion matrix to measure our system's efficiency in detail. Additionally, the LIME algorithm was used to visualize CNN's decision-making strategy.