Please wait ...
0% Complete
Home
/
11th International Conference on Computer and Knowledge Engineering
Semi-automatic Detection of Persian Stopwords using FastText Library
Authors :
Mohammad Dehghani
1
Mohammad Manthouri
2
1- Tarbiat Modares University
2- Shahed University
Keywords :
stopword, natural language processing, fastText, word embedding, text mining
Abstract :
A stopword is a word that does not add much semantic information to the text that despite of its very high frequency. Stopwords include prepositions, conjunctions, and pronouns. One of the steps in natural language processing is to remove stopwords to reduce dataset size and process faster. In this study, a semi-automatic method for collecting the Persian language's stopwords is proposed. The proposed method lists the stopwords of each text depending on its subject. For this purpose, based on a corpus of news texts, the Inverse Document Frequency (IDF) weight in the text is calculated for each word and the stopwords candidates are determined. Then, using the fastText library, the vector of each word is obtained. In the next step, five neighbors are found for each vector. Next, by removing duplicate words, the final list of stopwords (1014 stopwords) is collected. The result of simulations show the accuracy of detecting stopwords by the k-nearest neighbor method is 94.6%.
Papers List
List of archived papers
Cardiology Disease Diagnosis by Analyzing Histological Microscopic Images Using Deep Learning
Maria Salehpanah - Jafar Tanha - Zahra Jafari - SeyedEhsan Roshan - Sajad Rezaei
SGFL: A Federated Learning Approach for Non-IID Data Using Semi-Supervised DCGAN
Alireza Rabiee - Abolfazl Ajdarloo - Mohsen Rahmani
Efficient T-Count Fault-tolerant Quantum Clifford+T Multiplexer
Negin Mashayekhi - Shekoofeh Moghimi - Mohammad Reza Reshadinezhad
Analysis of Insect-plant Interactions Affected by Mining operations, A Graph Mining Approach
Mohammad Heydari - Ali Bayat - Amir Albadvi
Segmentation of Coronary Artery Stenosis in X-ray Angiography using Mamba Models
Fatemeh Fouladi - Ali Rostami - Hedieh Sajedi
An Effective Connectomics Approach for Diagnosing ADHD using Eyes-open Resting-state MEG
Nastaran Hamedi - Ali Khadem - Sajjad Vardast - Mehdi Delrobaei - Abbas Babajani-Feremi
A Genetic-based Fusion Approach of Persian and Universal Phonetic results for Spoken Language Identification
Ashkan Moradi - Yasser Shekofteh - Saeed Zarei
A Smart Electrochemical Biosensor for Arsenic Detection in Water
Keyvan Asefpour Vakilian
DTranIDS: A Two-Tiered Intrusion Detection System for RPL-based IoT Networks based on Decision Tree and Transformer Models
Mohammad Fazeli - Mohsen Raji - Mohammad Mahdi Fazeli
LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4
Mobina Kashaniyan - Amirhossein Ghassemi - Nasser Mozayani
more
Samin Hamayesh - Version 46.0.2