Context-Aware 3D Visualization of the Dynamic Environment

We present a graphical interface that provides context-dependent information about possible interactions with a complex cluttered environment, which can be used as a video overlay system in Head Mounted Devices (HMDs) in Augmented Reality applications. The system identifies task relevant objects in a cluttered scene, learns the typical interaction patterns with them in a teaching phase, and supports the user with information about the object functionality and handling in the interaction phase. The underlying system is capable of identification of objects in cluttered environment, providing information about possible task-dependent grasps, tracking object motion with high frame-rate and prediction of possible actions from a learned action graph that represents possible handling variants for a specific object. The system can be used to train a novice in an unknown environment providing expert knowledge. We present our implementation of the system in a service scenario of a kitchen environment, where the system supports the user while performing the actions.