International Conference on Computer and Knowledge Engineering

Home / 12th International Conference on Computer and Knowledge Engineering

Introducing E4MT and LMBNC: Persian pre-processing utilities

Authors :

Zakieh Shakeri¹ Mehran Ziabary² Behrooz Vedadian³ Fatemeh Azadi⁴ Saeed Torabzadeh⁵ Arian Atefi⁶

1- Targoman Intelligent Processing Co. Pjc. 2- Targoman Intelligent Processing Co. Pjc. 3- Targoman Intelligent Processing Co. Pjc. 4- Targoman Intelligent Processing Co. Pjc. 5- Amirkabir university of technology 6- Targoman Intelligent Processing Co. Pjc.

Keywords :

Natural language processing،Neural Machine Translation،Persian text pre-processing

Abstract :

In this paper, we introduce two utilities, extensively used in our services and products. A Persian pre-processor(E4MT) we use for both training and inference in our machine translation services and a corpora-level language model-based error corrector(LMBNC), which we apply to corpora before training. E4MT(Essential tools for MT) consists of character normalization, spell correction, entity tagging, and tokenization/detokenization modules. It handles the Persian large vocabulary size problem by approximately reducing the vocabulary size by a factor of 2. We show that applying E4MT on the English-Persian translation task, yields an improvement of at least 1.2 BLEU over other toolkits. We apply LMBNC on the training corpora, which uses a domain-specific language model to identify context-dependent misspellings. The results show, using this corrected training corpora improves the English-Persian translation quality by 0.6 BLEU over its baseline. Additionally, the manual evaluation shows 97.9\% precision for E4MT and 98.1\% precision for LMBNC.