TSA2: Temporal Segment Adaptation and Aggregation for Video Harmonization

Video composition merges the foreground and background of different videos, presenting challenges due to variations in capture conditions (e.g., saturation, brightness, and contrast). Video harmonization is a vital process in achieving a realistic composite by seamlessly adjusting the foreground’s appearance to match the background. In this paper, we propose TSA
²
, a novel method for video harmonization that incorporates temporal segment adaptation and aggregation. TSA
²
divides the inharmonious input sequence into temporal segments, each corresponding to a different frame rate, allowing effective utilization of complementary information within each segment. The method includes the Temporal Segment Adaptation module, which learns and remaps the distribution difference between background and foreground regions, and the Temporal Segment Aggregation module, which emphasizes and aggregates cross-segment information through element-wise correlations. Experimental results demonstrate that TSA
²
outperforms advanced image and video harmonization methods quantitatively and qualitatively.