Evaluation of an Automatic Question Generation Approach Using Ontologies

Advancements in Semantic Web techniques have led to the emergence of ontology based question generation. Ontologies are used to represent domain knowledge in the form of concepts, instances and their relationships as their elements. Many research strategies for generating questions using ontologies have been proposed but little work has been done on investigating whether an ontology is an appropriate source of data for question generation. Since there is no standard guideline for developing an ontology, the representation of ontology elements might vary in many ways, and this paper aims to investigate how the naming of ontology elements would affect the questions generated. In order to achieve this aim, two research questions will be investigated which are: how many correct questions can be generated from an ontology, and what are the reasons for incorrect questions being generated. Categorized question templates and a set of question strategies for mapping templates with a concept in an ontology are proposed. A prototype has been developed with a Reader to read data from input file and 3 question generators namely termQG, ClassQG and PropertyQG to generate questions for 3 ontological approaches. After questions have been generated, the number of correct questions generated is calculated and the reasons for incorrect questions are identified. Two ontologies have been used, an Operating System Ontology and a Travel Ontology. Twenty question templates from three question categories - definition, concept completion and comparison - together with 5 question generation strategies have been used in this evaluation. Results shows that more than half of the questions generated are correct and there are 3 distinct reasons why incorrect questions may be generated. The main contribution to incorrect question generation was inappropriate naming of ontology elements where 4 distinct categories are further identified. In addition, evaluation shows that the object type should be considered when designing question templates. Furthermore, the evaluation indirectly shows the effectiveness of the ontological approaches for generating questions from a real-world ontology.