A bi-directional facial attribute transfer framework: transfer your single facial attribute to a portrait illustration

Facial attribute transfer aims to transfer target facial attributes-such as beard, bangs and opening mouth-to a face without them in a source facial image while keeping non-target attributes of the face intact. Existing methods for facial attribute transfer are basically homogeneous images oriented, which focus on transferring target attributes to (or between) photorealistic facial images. In this paper, facial attribute transfer between heterogeneous images is addressed, which is a new and more challenging task. More specifically, we propose a bi-directional facial attribute transfer method based on GAN (generative adversarial network) and latent representation in a new way, for the instance based facial attribute transfer that aims to transfer a target facial attribute with its basic shape from a reference photorealistic facial image to a source realistic portrait illustration and vice versa (i.e., erasing the target attribute in the facial image). How to achieve visual style consistency of the transferred attribute in the heterogeneous result images and overcome information dimensionality imbalance between photorealistic facial images and realistic portrait illustrations are the key points in our work. We deal with content and visual style of an image separately in latent representation learning by the composite encoder designed with the architecture of convolutional neural network and fully connected neural network, which is different from previous latent representation based facial attribute transfer methods that mix content and visual style in a latent representation. The approach turns out to well preserve the visual style consistency. Besides, we introduce different multipliers for weights of loss items in our objective functions to balance information imbalance between heterogeneous images. Experiments show that our method is capable of achieving facial attribute transfer between heterogeneous images with good results. For purpose of quantitative analysis, FID scores of our method on a couple of datasets are also given to show its effectiveness.