THE JIPDEC CHECKLIST-BASED GUIDELINE FOR EXPERT-SYSTEM EVALUATION

Recent advances in and the proliferation of AI technologies make it essential to evaluate expert systems (ESs). There are three criteria for evaluation: technical, economic, and social. In order to focus on the technical side-verification, validation, and testing methods-the Japan Information Processing Development Center (JIPDEC) set up a research committee and started a 3-year project (1989-1991) to develop a guideline for expert system evaluation. The committee was composed of researchers from computer and tool retailers, representative users and from universities, as these groups of people have wide experience in developing and evaluating ESs and AI tools. This was the first large-scale precompetitive study project of its kind in Japan. To develop the guideline, we must determine the class of ESs, the life cycle model, the salient evaluation items, and the methodology of ES evaluation. This article describes the basic principles and overview of the project. First, we identify a class of expert systems to be critically evaluated. Second, we define an ES development life cycle model based on both rapid prototyping and phased approaches. Third, we propose a checklist for ES evaluation to cover the ES class and the life cycle model. Fourth, we present a framework for using the checklist in the development of ESs. And fifth, we validate the guideline itself by applying it to the development processes of four representative ESs. (C) 1994 John Wiley & Sons, Inc.