Scaling the ISLE Framework: Use of Existing Corpus Resources for Validation of MT Evaluation Metrics across Languages

Abstract

This paper describes the next step in a machine translation (MT) evaluation (MTE) research program previously reported on at MT Summit 2001. The development of this evaluation methodology has benefited from the availability of two collections of source language texts and the results of processing these texts with several consumer off-the-shelf (COTS) MT engines (DARPA 1994, Doyon, Taylor, & White 1999). The crucial characteristic of this methodology is a systematic development of a predictive relationship between discrete, well-defined metrics (a set of quality test scores) and specific information processing tasks that can be reliably performed with output of a given MT system. One might view the intended outcomes as (1) a system for classifying MT output in terms of the information processing functions it can serve and (2) an indicator for research and development directions in MT designed to serve a specific information processing function.

Resources

Details