Marginal Contrastive Correspondence for Guided Image Generation

Exemplar-based image translation establishes dense correspondences between a conditional input and an exemplar (from two different domains) for leveraging detailed exemplar styles to achieve realistic image translation. Existing work builds the cross-domain correspondences implicitly by minimizing feature- wise distances across the two domains. Without explicit exploitation of domain-invariant features, this approach may not reduce the domain gap effectively which often leads to sub-optimal correspon-dences and image translation. We design a Marginal Contrastive Learning Network (MCL-Net) that explores contrastive learning to learn domain-invariant features for realistic exemplar-based image translation. Specifically, we design an innovative marginal contrastive loss that guides to establish dense correspondences explicitly. Nevertheless, building correspondence with domain-invariant semantics alone may impair the texture patterns and lead to degraded texture generation. We thus design a Self-Correlation Map (SCM) that incorporates scene structures as auxiliary information which improves the built correspondences substantially. Quantitative and qualitative experiments on multifarious image translation tasks show that the proposed method outperforms the state-of-the-art consistently.

1. Introduction

Image-to-image translation refers to the image generation conditioned on the certain inputs from other domains [25], [29], [32], and has delivered impressive advances with the emergence of Generative Adversarial Networks (GAN s) [5] in recent years. As a typical ill-posed problem, diverse solutions are naturally allowed in image translation tasks as one conditional input could corresponds to multiple image instances. Faithful control of the generation style not only enables diverse generation given certain conditions but also flexible user control of the desired generation. However, yielding high-fidelity images with controllable styles still remains a grand challenge. Figure 1.

Learning domain-invariant features for building correspondence across domains: we exploit contrastive learning between the conditional input and the ground truth by pulling features at the same position closer while pushing those at different positions apart. With the learned condition encoder and image encoder, explicit feature correspondences can be built up between the conditional input and the exemplar image.