Enhancing Financial Security: A Multimodal Approach to User Authentication

The Digital India initiative not only raises awareness about the benefits of digitization but also underscores the importance of a secure authentication system to combat online fraud. In this study, we propose a multimodal approach integrating audio and image modalities, along with real-time lip-sync detection, to address the issue of unauthorized account usage. The combined utilization of these modalities proves beneficial, particularly in regions with poor internet connectivity. The decision to grant access to a specific individual is dependent upon the combined score derived from both modalities. Furthermore, to enhance security, real-time lip-sync detection (with the help of a synchronization module) is employed to ensure that no prerecorded image or audio is presented during system login access. The synchronization model outperforms the LRS2 dataset and is used for multimodal verification tasks on the GRID dataset with corrupted data to give an EER of 1.44 %.