Interactive and Markerless Visual Recognition of Brazilian Sign Language Alphabet

The automatic recognition of sign languages will increase the inclusion of non-verbal persons in society by allowing them to communicate with people who are not familiar with sign language. To this end, recently some systems have been proposed to automatically recognize sign language. Among them, those with external aids such as gloves, markers, clothing/background control, or radars usually have high accuracy, but are not practical in daily life situations. On the other hand, those that use only a camera, such as prevalent smartphone cameras, are practical but have lower accuracy. In this work, we present a system that can recognize the alphabet in the Brazilian sign language, Língua Brasileira de Sinais (LIBRAS), using only a camera yet achieving high accuracy. Like most existing works, our system captures and processes the image of hand gesture and uses a classifier to recognize the sign. However, unlike existing works, the classifier is the Inception-v3 neural network trained with transfer learning on our custom-collected LIBRAS alphabet dataset. Performance evaluations show the system recognizes LIBRAS alphabet with 97% accuracy. We also developed an interactive app, demonstrating that it can run in real time.