International Conference on Computer and Knowledge Engineering

Home / 11th International Conference on Computer and Knowledge Engineering

Semi-automatic Detection of Persian Stopwords using FastText Library

Authors :

Mohammad Dehghani¹ Mohammad Manthouri²

1- Tarbiat Modares University 2- Shahed University

Keywords :

stopword, natural language processing, fastText, word embedding, text mining

Abstract :

A stopword is a word that does not add much semantic information to the text that despite of its very high frequency. Stopwords include prepositions, conjunctions, and pronouns. One of the steps in natural language processing is to remove stopwords to reduce dataset size and process faster. In this study, a semi-automatic method for collecting the Persian language's stopwords is proposed. The proposed method lists the stopwords of each text depending on its subject. For this purpose, based on a corpus of news texts, the Inverse Document Frequency (IDF) weight in the text is calculated for each word and the stopwords candidates are determined. Then, using the fastText library, the vector of each word is obtained. In the next step, five neighbors are found for each vector. Next, by removing duplicate words, the final list of stopwords (1014 stopwords) is collected. The result of simulations show the accuracy of detecting stopwords by the k-nearest neighbor method is 94.6%.

Papers List

List of archived papers

Improving LoRaWAN Scalability for IoT Applications using Context Information

Hamed Mahmoudi - Behrouz ShahgholiGhahfarokhi

Improved TrustChain for Lightweight Devices

Seyed Salar Ghazi - Haleh Amintoosi

VVC-AAR: Adaptive Attention-Aware Resolution and Residual Coding for Perceptually Optimized Ultra-Low Bitrate VVC Compression

Yaghoub Saberi - Somayeh Arab Najafabadi - Mohammadreza Hemmati

Age Estimation Based on Facial Images Using Hybrid Features and Particle Swarm Optimization

NILOUFAR MEHRABI - SAYED PEDRAM HAERI BOROUJENI

LightFedSelect: A Lightweight Framework for Byzantine-Robust Federated Learning

Seyed Saeed Razavi - Seyed Arsalan Vasegh Rahim Parvar - Soroosh Dadashi Pakdeh - Mohammad Matin Rezaeifard - Morteza Mollaie Chafi - Reza Ebrahimi Atani

Adaptive Channel Estimation for MIMO-OFDM Systems in Impulsive Noise Environments

Mojtaba Hajiabadi

Using Deep Learning for Classification of Lung Cancer on CT Images in Ardabil Province

Mohammad Ali Javadzadeh Barzaki - Jafar Abdollahi - Mohammad Negaresh - Maryam Salimi - Hadi Zolfeghari - Mohsen Mohammadi - Asma Salmani - Rona Jannati - Firouz Amani

An Efficient Approach for Breast Abnormality Detection through High-Level Features of Thermography Images

Farhad Abedinzadeh Torghabeh - Yeganeh Modaresnia - Seyyed Abed Hosseini

Real-Time Forecasting Using Mixed Frequency Time-Series Data

Armin Khayati - Mohammad Taheri - Koorush Ziarati

An Exploratory Study of the Relationship between SATD and Other Software Development Activities

Shima Esfandiari - Ashkan Sami