Sentence Level Language Identification in Code-mix Gujarati Language with Transformers

Code-mix is gaining popularity due to its widespread usage on social media platforms. Even bilingual users converse in code-mix in their day-to-day life due to the popularity of the English language mix with the user’s mother tongue. Codemix phenomena come under NLP(Natural Language Processing), again a subpart of Artificial Intelligence. Code-mix language means grammatically following one language but has used another language’s words. Identification of code-mix is challenging due to using two languages in one sentence. Our motive is to identify the code-mix language at the sentence level. Already machine learning algorithms have been applied to identify sentence-level code-mix languages. In this article, we have applied Deep learning and a transformer-based framework for the sentence-level language identification task. We have used BERT in the Gujarati code-mix data set and received 96.2%, which is a more accurate result than the machine learning and Deep learning algorithms have achieved.