International Conference on Computer and Knowledge Engineering

Home / 11th International Conference on Computer and Knowledge Engineering

Semi-automatic Detection of Persian Stopwords using FastText Library

Authors :

Mohammad Dehghani¹ Mohammad Manthouri²

1- Tarbiat Modares University 2- Shahed University

Keywords :

stopword, natural language processing, fastText, word embedding, text mining

Abstract :

A stopword is a word that does not add much semantic information to the text that despite of its very high frequency. Stopwords include prepositions, conjunctions, and pronouns. One of the steps in natural language processing is to remove stopwords to reduce dataset size and process faster. In this study, a semi-automatic method for collecting the Persian language's stopwords is proposed. The proposed method lists the stopwords of each text depending on its subject. For this purpose, based on a corpus of news texts, the Inverse Document Frequency (IDF) weight in the text is calculated for each word and the stopwords candidates are determined. Then, using the fastText library, the vector of each word is obtained. In the next step, five neighbors are found for each vector. Next, by removing duplicate words, the final list of stopwords (1014 stopwords) is collected. The result of simulations show the accuracy of detecting stopwords by the k-nearest neighbor method is 94.6%.

List of archived papers

FarCQA: A Farsi Community Dataset for Question Classification and Answer Selection

Saba Emami - Maedeh Mosharraf

Damage Detection After the Earthquake Using Sentinel-1 and 2 Images and Machine Learning Algorithms (Case Study: Sarpol-e Zahab Earthquake)

Niloofar Alizadeh - Behnam Asghari Beirami - Mehdi Mokhtarzade

Enhanced Principal-curve based Classifiers for Time-series Label Prediction

Seyed Aref Hakimzadeh - Koorush Ziarati

Designing a High Perfomance and High Profit P2P Energy Trading System Using a Consortium Blockchain Network

Poonia Taheri Makhsoos - Behnam Bahrak - Fattaneh Taghiyareh

A Survey of the AVOA Metaheuristic Algorithm and its Suitability for Power System Optimization and Damping Controller Design

Aliyu Sabo - Theophilus Ebuka Odoh - Samuel Habu - Hossien Shahinzadeh - Farshad Ebrahimi

An overview of Business Intelligence research in healthcare organizations using a topic modeling approach

Mohammad Mehraeen - Laya Mahmoudi - Mohammad Hossein Sharifi

Improve the utility of tensor cores by compacting sparse matrix technique

Mohammad.S Abazari - Mahsa Zahedi - Abdorreza Savadi

Optimizing Question-Answering Framework Through Integration of Text Summarization Model and Third-Generation Generative Pre-Trained Transformer

Ervin Gubin Moung - Toh Sin Tong - Maisarah Mohd Sufian - Valentino Liaw - Ali Farzamnia - Farashazillah Yahya

Optimizing MR Image Registration for Accurate Brain Volume Measurement in Children with Autism Spectrum Disorder

Shiva Sanati - Mahdi Saadatmand

Enhanced Atrial Fibrillation (AF) Detection via Data Augmentation with Diffusion Model

Arash Vashagh - Amirhossein Akhoondkazemi - Sayed Jalal Zahabi - Davood Shafie

more

Samin Hamayesh - Version 41.3.1