Understanding Vehicle Interaction in Driving Video with Spatial-temporal Deep Learning Network

This paper develops a deep learning method to identify vehicle interactions from driving video and creates a dataset for network training and testing. This is an original work targeting qualitative vehicle interactions from continuous visual sensing of image velocity, position, and vehicle width/depth. The motion profile is obtained from driving video as a subspace for learning and classification. A spatial-temporal 2D CNN Network (named ST2CN) scans streaming frames to detect the vehicle status at every frame without latency from a short history. The motion continuity and temporal dependency are well preserved by this ST2CN based network. The labeled interaction dataset has 300-minute videos sampled at dense rate to include 13 classes of interactions. Our initial results serve as a benchmark for sensing vehicle interactions. The detection of surrounding vehicle's action helps qualitative planning and smooth driving of intelligent vehicles.