Task-oriented Dependency Parsing Evaluation Methodology

Traditional parser evaluation with attachment scores is not helpful for researchers who want to find the most suitable parser for their application. First, because it is being done for a domain which is almost always different from the domain of the application and second because many of the tested dependencies are irrelevant for the application. The alternative extrinsic evaluation is problematic as well, since it is difficult to find a suitable data set and because it is not straightforward how to measure the quality of the parser in the context of a broader appllication. We propose a method which combines the strengths of attachment scores and extrinsic evaluation and avoids their weaknesses. On the one hand we use the very robust attachment scores, We apply our approach to RTE-7 data in order to demonstrate how it works.