Verify-Pro: A Framework for Server Authentication using Communication Protocol Dialects

Customizing program binary and communication features is a commonly adopted strategy to counter network security threats like session hijacking, context confusion, and impersonation attacks. A potential attacker may have enough time to launch an attack targeting these vulnerabilities by rerouting the target request to a malicious server or hijacking the traffic. This paper presents a novel system Verify-Pro, a framework for server authentication using communication protocol dialects, to customize the communication features, enforce continuous authentication, detect the adversary, and prevent sensitive information leakage. Specifically, we leverage a machine learning approach (pre-trained neural network model) on both client and server machines to trigger a specific dialect that dynamically changes for each request (e.g., get filename in FTP). Then, a decision tree algorithm is developed to automatically detect the adversary and terminate the entire session if the message is from an adversary. We implement a prototype of Verify- Pro and evaluate its practicality on standard communication protocol: FTP (File Transfer Protocol) and present a case study of the internet of things protocol MQTT (Message Queuing Telemetry Transport). Our experimental results show that by sending misleading information through message packets from an attacker at the application layer, the recipient can identify whether the sender is genuine or spoofed, with a negligible overhead of < 1%.