International Conference on Computer and Knowledge Engineering

Home / 12th International Conference on Computer and Knowledge Engineering

SAT Based Analogy Evaluation Framework For Persian Word Embeddings

Authors :

Seyed Ehsan Mahmoudi¹ Mehrnoush Shamsfard²

1- Computer Science and Engineering Department, Shahid Beheshti University, Tehran, Iran 2- Faculty of Computer Science and Engineering, Shahid Beheshti University

Keywords :

Analogy test،Semantic Similarity،Embedding Evaluation،Persian Langugae Processing

Abstract :

In recent years there has been a special interest in word embeddings as an approach to convert words to vectors. It has been a focal point to understand how much of the semantics of the words has been transferred into embedding vectors. Intrinsic evaluation of word embeddings is cheaper than evaluating them extrinsically and it is usually costly to evaluate the downstream application end-to-end in order to determine the quality of the used embedding model. Generally the word embeddings are evaluated through a number of tests, including analogy test. In this paper we propose a test framework for Persian embedding models. Persian is a low resource language and there is no rich semantic benchmark to evaluate word embedding models for this language. In this paper we introduce an evaluation framework including a hand crafted Persian SAT based analogy dataset, a colliquial test set (specific to Persian) and a benchmark to study the impact of various parameters on the semantic evaluation task.