Augmenting moving planar surfaces robustly with video projection and direct image alignment

Augmented reality applications based on video projection, to be effective, must track moving targets and make sure that the display remains aligned even when they move, but the projection can severely alter their appearances to the point where traditional computer vision algorithms fail. Current solutions consider the displayed content as interference and largely depend on channels orthogonal to visible light. They cannot directly align projector images with real-world surfaces, even though this may be the actual goal. We propose instead to model the light emitted by projectors and reflected into cameras and to consider the displayed content as additional information useful for direct alignment. Using a color camera, our implemented software successfully tracks with subpixel accuracy a planar surface of diffuse reflectance properties at an average of eight frames per second on commodity hardware, providing a solid base for future enhancements.