A multimodal language to communicate with life supporting robots through a touch screen and a speech interface

This paper proposes a multimodal language to communicate with life supporting robots through a touch screen and a speech interface. The language is designed for untrained users who need support in daily lives from cost effective robots. In this language, the users can combine spoken and pointing messages in an interactive manner in order to convey their intensions to robots. Spoken messages include verb and noun phrases which describe intensions. Pointing messages are given when users finger-touch a camera image, a picture containing a robot body, or a button on a touch screen at hand, which convey a location in their environment, a direction, a body part of the robot, a cue, a reply to a query, or other information to help the robot. This work presents the philosophy and structure of the language.