Steps in Identifying Interaction Design Patterns for Multimodal Systems

The context of this work is usability engineering for mutimodal interaction. In contrast to other work that concentrates on prototyping toolkits or abstract guidelines, this research focuses on user interface patterns for multimodal interaction. Designing multimodal applications requires several skills ranging from design and implementation. Thus, different kinds of patterns (from architecture patterns to user interface patterns) can be applied to this field. This work focuses on user-task near user interface patterns. At first, a traditional approach of modality selection based on task- and context-based rules is presented. Next, a twofold process of pattern mining is presented. In the first phase, pattern candidates are derived top-down from proven knowledge about how multimodality enhances usability. In the second phase, literature is mined for real solutions to underpin these pattern candidates and find new ones. Along with this, relationships between patterns are depicted.