International Conference on Computer and Knowledge Engineering

Home / 12th International Conference on Computer and Knowledge Engineering

Introducing E4MT and LMBNC: Persian pre-processing utilities

Authors :

Zakieh Shakeri¹ Mehran Ziabary² Behrooz Vedadian³ Fatemeh Azadi⁴ Saeed Torabzadeh⁵ Arian Atefi⁶

1- Targoman Intelligent Processing Co. Pjc. 2- Targoman Intelligent Processing Co. Pjc. 3- Targoman Intelligent Processing Co. Pjc. 4- Targoman Intelligent Processing Co. Pjc. 5- Amirkabir university of technology 6- Targoman Intelligent Processing Co. Pjc.

Keywords :

Natural language processing،Neural Machine Translation،Persian text pre-processing

Abstract :

In this paper, we introduce two utilities, extensively used in our services and products. A Persian pre-processor(E4MT) we use for both training and inference in our machine translation services and a corpora-level language model-based error corrector(LMBNC), which we apply to corpora before training. E4MT(Essential tools for MT) consists of character normalization, spell correction, entity tagging, and tokenization/detokenization modules. It handles the Persian large vocabulary size problem by approximately reducing the vocabulary size by a factor of 2. We show that applying E4MT on the English-Persian translation task, yields an improvement of at least 1.2 BLEU over other toolkits. We apply LMBNC on the training corpora, which uses a domain-specific language model to identify context-dependent misspellings. The results show, using this corrected training corpora improves the English-Persian translation quality by 0.6 BLEU over its baseline. Additionally, the manual evaluation shows 97.9\% precision for E4MT and 98.1\% precision for LMBNC.

Papers List

List of archived papers

Enhanced Atrial Fibrillation (AF) Detection via Data Augmentation with Diffusion Model

Arash Vashagh - Amirhossein Akhoondkazemi - Sayed Jalal Zahabi - Davood Shafie

Efficient Sub-Carrier Relationship Extraction for Human Activity Recognition via EEGNet in Wireless Sensing

Siavash Zaravashan - Sadegh ArefiZadeh - Sajjad Torabi

Towards Low-Overhead Mitigation of Trojan Bit-Flip Attacks on DNNs via Causal Inference

Bahare Gholami - Mohsen Raji

Optimization of quantum secret sharing communication using corresponding bits

Mahsa Khorrampanah - Mohammad Bolokian - Monireh Houshmand

Introducing Meta-Contrastive Adaptive Autoencoder to Tackle Cold-Start Challenges in Sparse Domains

Hossein Rashid - Erfan Arzhmand - Fatemeh Hosseini

LightFedSelect: A Lightweight Framework for Byzantine-Robust Federated Learning

Seyed Saeed Razavi - Seyed Arsalan Vasegh Rahim Parvar - Soroosh Dadashi Pakdeh - Mohammad Matin Rezaeifard - Morteza Mollaie Chafi - Reza Ebrahimi Atani

Deep Learning Feature Extraction for COVID-19 Detection Algorithm using Computerized Tomography Scan

Maisarah Mohd Sufian - Ervin Gubin Moung - Chong Joon Hou - Ali Farzamnia

LLM-Driven AutoML for Cross-Lingual Handwritten OCR: Closed-Loop Neural Architecture Search with GPT-5, GPT-4o, and Claude Sonnet 4

Mobina Kashaniyan - Amirhossein Ghassemi - Nasser Mozayani

Facial Emotion Recognition Under Mask Coverage Using a Data Augmentation Technique

Aref Farhadipour - Pouya Taghipour

Prediction of rTMS Treatment Response in Depression Using a Frequency-Based EEG Biomarker

Ali Asadi Zeidabadi - Saeid Rashidi