Predictive and objective evaluation of speech understanding: the “challenge” evaluation campaign of the I3 speech workgroup of the French CNRS

Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002)

Abstract

This paper presents a new paradigm of "challenge" evaluation of Spoken Language Understanding. This methodology aims at a quantitative assessment with a high diagnostic power, by opposition with standard ATIS-like frameworks. This paper details the methodology as well as the results of an evaluation campaign held by the French CNRS research agency. The benefits of this methodology are also discussed.