VISA: Video Interactive Search with Advanced Visual Programming

Video retrieval has become an important task in computer vision, with video content uploaded to the Internet every hour. Along with retrieving the relevant visual content, users may also want to perform several post-processing steps such as visual editing, understanding or video summarizing. However, to our knowledge, there is no such integrated system that enables users to perform downstream visual understanding and editing tasks via text prompts. In this work, we propose VISA framework, which combines a visual programming module with a video search system. Specifically, our interactive framework offers fundamental video retrieval with semantic search, text search and audio search with descriptive inputs summarized by a large language model (LLM). After obtaining the video frame results, users can provide natural language instructions as guidance for image understanding and editing tasks. Having the in-context learning capability of LLMs, our visual programming module generates high-level and interpretable pseudocodes from the given instructions. The corresponding Python programs are then executed to achieve the desired results. We evaluate our VISA framework on the 2023 Ho Chi Minh City AI Challenge dataset and the image editing component on the MagicBrush benchmark.