Two-View Correspondence Learning With Local Consensus Transformer

Correspondence learning is a crucial component in multiview geometry and computer vision. The presence of heavy outliers (mismatches) consistently renders the matching problem to be highly challenging. In this article, we revisit the benefits of local consensus (LC) in traditional feature matching and introduce the concept of LC to design a trainable neural network capable of capturing the underlying correspondences. This network is named the LC transformer (LCT) and is specifically tailored for wide-baseline stereo applications. Our network architecture comprises three distinct operations. To establish the neighbor topology, we employ a dynamic graph-based embedding layer as the initial step. Subsequently, these local topologies serve as guidance for the multihead self-attention layer, enabling it to extract a more extensive contextual understanding through channel attention (CA). Following this, order-aware graph pooling is applied to extract the global context information from the embedded LC. Through the experimental analysis, the ablation study reveals that PointNet-like learning models can, indeed, benefit from the incorporation of LC. The proposed model achieves state-of-the-art performance in both challenging scenes, namely, the YFCC100M outdoor and SUN3D indoor environments, even in the presence of more than 90% outliers.