Image-based view synthesis for enhanced perception in teleoperation

The teleoperation of equipment under impoverished sensing and communication delays, cannot be handled efficiently by conventional remote control techniques. Our approach to this problem is based on an augmented reality control mode in which a graphical model of the equipment is overlaid upon real views from the work-site. A basic capability required in order to produce such an augmented reality mode is the ability to synthesize visual information from new viewpoints based upon existing ones, so as to compensate for the sparsity of real data. Our approach to the problem of image-based view synthesis is based upon the implicit construction of a 3D approximation of the scene, composed of planar triangular patches. New views are then generated by texture-mapping the available real image data onto the reprojected triangles. Ln order to generate a physically valid joint-triangulation which minimizes the distortions in the rendering of the new view, an iterative approach is utilized. This approach begins with an initial triangulation and refines it iteratively through node-linking alterations and a split and merge process, based upon correlation values between corresponding triangular patches. The paper presents results for both synthetic and real scenes.