VADER: Video Alignment Differencing and Retrieval

We propose VADER, a spatio- temporal matching, alignment, and change summarization method to help fight misinformation spread via manipulated videos. VADER matches and coarsely aligns partial video fragments to candidate videos using a robust visual descriptor and scalable search over adaptively chunked video content. A transformer- based alignment module then refines the temporal localization of the query fragment within the matched video. A space- time comparator module identifies regions of manipulation between aligned content, invariant to any changes due to any residual temporal misalignments or artifacts arising from non- editorial changes of the content. Robustly matching video to a trusted source enables conclusions to be drawn on video provenance, enabling informed trust decisions on content encountered. Code and data are available at https://github.com/AlexBlck/vader

1. Introduction

Fake news and misinformation are major societal problems, much of it caused by manipulated videos shared to misrepresent events. Detecting manipulation presents only a partial solution; most edited content is not misinformation [22]. Many emerging solutions, therefore, focus on provenance - determining the origins of content, what was changed, and by whom [43], [6]. Securely embedding such an audit trail inside video metadata can help consumers make better trust decisions on content, and is the basis of recent open standards (e.g. C2PA) [11], [1]. However, such metadata can be trivially stripped or replaced.