Please wait ...
0% Complete
Home
/
12th International Conference on Computer and Knowledge Engineering
Introducing E4MT and LMBNC: Persian pre-processing utilities
Authors :
Zakieh Shakeri
1
Mehran Ziabary
2
Behrooz Vedadian
3
Fatemeh Azadi
4
Saeed Torabzadeh
5
Arian Atefi
6
1- Targoman Intelligent Processing Co. Pjc.
2- Targoman Intelligent Processing Co. Pjc.
3- Targoman Intelligent Processing Co. Pjc.
4- Targoman Intelligent Processing Co. Pjc.
5- Amirkabir university of technology
6- Targoman Intelligent Processing Co. Pjc.
Keywords :
Natural language processing،Neural Machine Translation،Persian text pre-processing
Abstract :
In this paper, we introduce two utilities, extensively used in our services and products. A Persian pre-processor(E4MT) we use for both training and inference in our machine translation services and a corpora-level language model-based error corrector(LMBNC), which we apply to corpora before training. E4MT(Essential tools for MT) consists of character normalization, spell correction, entity tagging, and tokenization/detokenization modules. It handles the Persian large vocabulary size problem by approximately reducing the vocabulary size by a factor of 2. We show that applying E4MT on the English-Persian translation task, yields an improvement of at least 1.2 BLEU over other toolkits. We apply LMBNC on the training corpora, which uses a domain-specific language model to identify context-dependent misspellings. The results show, using this corrected training corpora improves the English-Persian translation quality by 0.6 BLEU over its baseline. Additionally, the manual evaluation shows 97.9\% precision for E4MT and 98.1\% precision for LMBNC.
Papers List
List of archived papers
An Energy-efficient Clustering Method based on Butterfly Optimization Algorithm by Considering the Criterion of Intra-cluster Distances in WSNs
Fariba Saghi Hadi S. Aghdasi
Area-Efficient VLSI Implementation of Bit-Serial Multiplier Using Polynomial Basis over GF(2m)
Saeideh Nabipour - Javad Javidan - Gholamreza Zare Fatin
An Efficient Planning Method for Autonomous Navigation of a Wheeled-Robot based on Deep Reinforcement Learning
Ali Salimi Sadr - Mahdi Shahbazi Khojasteh - Hamed Malek - Armin Salimi-Badr
Solving the influence maximization problem by using entropy and weight of edges
Farzaneh Kazemzadeh - Amir Karian - Mitra Mirzarezaee - Ali Asghar Safaei
Trust Management Enhancement for the Internet of Things: a Smart Contract Approach
Amin Rouzbahani - Fattaneh Taghiyareh
A novel hybrid DMHS-GMDH algorithm to predict COVID-19 pandemic time series
Ahmad Taheri - Shahriar Ghashghaei - Amin Beheshti - Keyvan RahimiZadeh
Evolutionary Approach to GAN Hyperparameter Tuning: Minimizing Discriminator and Generator Loss Functions
Sajad Haghzad Klidbary - Anahita Babaei - Ramin Ghorbani
Underwater Image Super-Resolution using Generative Adversarial Network-based Model
Alireza Aghelan - Modjtaba Rouhani
A Facial Deepfake Detection Approach using CNN-based Models, Swin Transformer and Classifier Fusion
Alireza Honardoost - Mahdie Rahmati - Babak Nasersharif
Leveraging the Power of Object Detection Models in Identifying Litter for a Significant Reduction in Environmental Pollution
Lim Zhen Xian - Ervin Gubin Moung - Jason Teo Tze Wi - Nordin Saad - Farashazillah Yahya - Tiong Lin Rui - Ali Farzamnia
more
Samin Hamayesh - Version 46.0.2