International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Leveraging Self-Supervised Models for Automatic Whispered Speech Recognition

Authors :

Aref Farhadipour¹ Homa Asadi² Volker Dellwo³

1- University of Tehran 2- university of isfahan 3- Zurich University of Applied Sciences

Keywords :

Automatic Speech Recognition،whisper speech to text،self-supervised learning،speech processing،deep learning،transformers،Wavlm model،Dialect variation،Whisper model

Abstract :

In automatic speech recognition, any factor that alters the acoustic properties of speech can pose a challenge to the system's performance. This paper presents a novel approach for automatic whispered speech recognition in the Irish dialect using the self-supervised WavLM model. Conventional automatic speech recognition systems often fail to accurately recognise whispered speech due to its distinct acoustic properties and the scarcity of relevant training data. To address this challenge, we utilized a pre-trained WavLM model, fine-tuned with a combination of whispered and normal speech data from the wTIMIT and CHAINS datasets, which include the English language in Singaporean and Irish dialects, respectively. Our baseline evaluation with the OpenAI Whisper model highlighted its limitations, achieving a Word Error Rate (WER) of 18.8% and a Character Error Rate (CER) of 4.24% on whispered speech. In contrast, the proposed WavLM-based system significantly improved performance, achieving a WER of 9.22% and a CER of 2.59%. These results demonstrate the efficacy of our approach in recognising whispered speech and underscore the importance of tailored acoustic modeling for robust automatic speech recognition systems. This study provides valuable insights into developing effective automatic speech recognition solutions for challenging speech affected by whisper and dialect. The source codes for this paper are freely available.

Papers List

List of archived papers

Multi-Layered Defense Against Modern Phishing: A Dual-Sandbox and CDR Approach

Mahdi Seyfipoor - Mohammad Mahdi Eskandari

Multi-Task Transformer for Stock Market Trend Prediction

Seyed Morteza Mirjebreili - Ata Solouki - Hamidreza Soltanalizadeh - Mohammad Sabokrou

Enhancing Persian Word Sense Disambiguation with Large Language Models: Techniques and Applications

Fatemeh Zahra Arshia - Saeedeh Sadat Sadidpour

FedFog: A Serverless and Privacy-Aware Federated Learning Simulator for Edge–Fog Networks

Seyed Vahid Hashemi Nik - Seyed Mohammad Mahdi Asaadi - Somayeh Sobati-M

BioBERT-based SNP-traits Associations Extraction from Biomedical Literature

Mohammad Dehghani - Behrouz Bokharaeian - Zahra Yazdanparast

A Review on Secure Data Storage and Data Sharing Technics in Blockchain-based IoT Healthcare Systems

Seyedeh Somayeh Fatemi Nasab - Davoud Bahrepour - Seyed Reza Kamel Tabbakh

Adaptive Prioritization in Experience Replay Using Feedback from Multiple Learning Signals

Seyed Hossein Mostafavi - Mohammad Bagher Naghibi Sistani

Hardware-Efficient Pruned CNN Optimized by Neural Architecture Search and Genetic Algorithm for Diabetic Retinopathy Detection on STM32F746

Omid Askari Haddad - Sara Ershadi-Nasab

Using Deep Learning for Classification of Lung Cancer on CT Images in Ardabil Province

Mohammad Ali Javadzadeh Barzaki - Jafar Abdollahi - Mohammad Negaresh - Maryam Salimi - Hadi Zolfeghari - Mohsen Mohammadi - Asma Salmani - Rona Jannati - Firouz Amani

A parallel CNN-BiGRU network for short-term load forecasting in demand-side management

Arghavan Irankhah - Sahar Rezazadeh Saatlou - Mohammad Hossein Yaghmaee - Sara Ershadi-Nasab - Mohammad Alishahi