A Joint Learning-Based Method for Multi-View Depth Map Super Resolution

Depth map super resolution from multi-view depth or color images has long been explored. Multi-view stereo methods produce fine details at texture areas, and depth recordings would compensate when stereo doesn't work, e.g. at non-texture regions. However, resolution of depth maps from depth sensors are rather low. Our objective is to produce a high-res depth map by fusing different sensors from multiple views. In this paper we present a learning-based method, and infer a high-res depth map from our synthetic database by minimizing the proposed energy. As depth alone is not sufficient to describe geometry of the scene, we use additional features like normal and curvature, which are able to capture high-frequency details of the surface. Our optimization framework explores multi-view depth and color consistency, normal and curvature similarity between low-res input and the database and smoothness constraints on pixel-wise depth-color coherence as well as on patch borders. Experimental results on both synthetic and real data show that our method outperforms state-of-the-art.