Defining the syntax and semantics of natural visual languages

To bridge the gap between paper and electronic forms of documents, computers must be able to recognize and generate diagrams as well as text. Diagrams used in society are expressed in a variety of notations, which we call natural visual languages. Examples include notations used for mathematics, music, engineering drawings and architecture. These visual languages do not have fixed, formal definitions, but evolve through use in society. This paper examines the use of graph transformation in processing natural visual languages, describing the difficult problems in this domain, existing graph transformation work in this area, and competing methods. Many problems have not been adequately addressed by any technique. The use of graph transformation is appropriate, since the representation and manipulation of spatial and logical relationships is central to the computation.