Calibration between a 3D camera and an aerial information screen

We have formed an aerial screen with aerial imaging by retro reflection (AIRR) and detected gestures of a user with a 3D gesture recognition camera. This paper deals with a calibration problem between the coordinate systems of the aerial screen and the camera. We propose a calibration method of the 3D camera and the aerial information screen. We used a Kinect v2 for 3D gesture recognition. The Kinect detects positions of users' hands up to two users. We developed a calibration software and interactive applications by using Unity. In Unity, we prepared a camera for rendering, objects, and two squares that show hands' positions. The objects are displayed in the mid-air. Our developed program detects collision between one of the hands and one of the objects. The collision is determined after calibration. The calibration is automatically conducted by adjusting the view angle and the position of the camera in Unity to match the Kinect. In the calibration process, a user is instructed to place one of the hands on four corners of an aerial calibration pattern, one after another. The x-coordinate and y-coordinate of the hand position in the Kinect coordinate system are used to calibrate the central position in Unity. The z-coordinate is used to judge collision because it shows the distance from the hand to the aerial image. We have formed a 43-inch aerial screen. The aerial screen, tilting by 30 degrees, is floating at 35 cm above a reflective polarizer. We have realized aerial interaction.