GesSure: A Robust Face-Authentic Enabled Dynamic Gesture Recognition GUI Application


Ankit Jha, Ishita, Pratham G. Shenwai, Ayush Batra, Siddharth Kotian and Piyush Modi, Manipal Institute of Technology, MAHE, India


Using physical interactive devices like mouse and keyboards hinders naturalistic human-machine interaction and increases the probability of surface contact during a pandemic. Existing gesturerecognition systems do not possess user authentication, making them unreliable. Static gestures in current gesture-recognition technology introduce long adaptation periods and reduce user compatibility. Our technology places a strong emphasis on user recognition and safety. We use meaningful and relevant gestures for task operation, resulting in a better user experience. This paper aims to design a robust, faceverification-enabled gesture recognition system that utilizes a graphical user interface and primarily focuses on security through user recognition and authorization. The face model uses MTCNN and FaceNet to verify the user, and our LSTM-CNN architecture for gesture recognition, achieving an accuracy of 95% with five classes of gestures. The prototype developed through our research has successfully executed context-dependent tasks like save, print, control video-player operations and exit, and context-free operating system tasks like sleep, shut-down, and unlock intuitively. Our application and dataset are available as open source.


Gesture Recognition, Human Computer Interaction, Face Authentication, CNN-LSTM, MediaPipe.