Two-stream coupling network with bidirectional interaction between structure and texture for image inpainting

Since neural networks possess universal approximation property and the property can be a theoretical guarantee for image inpainting, the research, which uses deep learning technology to complete images, has attracted more attention from researchers in recent years. Given that structure and texture are two factors in the image, some existing studies have shown that modeling structure and texture can improve the effect of image inpainting. However, there exist close relationships between structure and texture, and they can contribute to each other, which is not considered in the existing works. This paper not only models the structure and texture of the image, but also constructs a bidirectional interactive relationship between them. A two-stream coupling network for image inpainting, which can effectively perform the bidirectional interaction between structure and texture, is proposed. Specifically, two parallel encoder-decoder networks are constructed to be responsible for structure reconstruction and texture synthesis respectively. Meanwhile, two interaction mechanisms called gated interaction unit (GIU) and spatial-channel attention (SCIM), are embedded between them, which perform bidirectional interaction to structure and texture information. GIU weighs the interaction between structure and texture features through gating mechanisms, and SCIM achieves a deeper interaction to the distant contexts of structure and texture through spatial-channel attention mechanisms. In this way, the network promotes both structure-guided texture synthesis and texture-facilitated structure reconstruction. Extensive experiments illustrate the superiority of the proposed method.