BaSa: A Technique to Identify Context based Common Tokens for Hindi Verses and Proses

Activating multilingual options on websites in this digital era has enabled the opportunity to generate text in different languages. Hindi has turned to be the significant language in different domains and mainly in the government domain. Proses (proses) and verses (verses) are generally used as an input data source to perform different operations like classification, clustering and so on. Carrying out different natural language processing (NLP) steps is a prerequisite for implementing machine learning algorithms. In this paper, the BaSa technique is proposed to identify context based common tokens from given corpora. Common tokens present at different levels of Natural language processing (NLP) are computed. It will give guidelines in the form for researchers working in Hindi NLP.Corpus of 820 proses and 710 verses is processed. It is observed that percentage of number of tokens reduced at each stage of NLP is almost same for proses and verses.