A New ITU-T Recommendation on the Evaluation of Telephone-Based Spoken Dialogue Systems

Abstract

This article describes efforts which have recently been undertaken by the International Telecommunication Union (ITU-T) to agree on common methods for evaluating telephone services based on spoken dialogue systems. As a result of these efforts, a new ITU-T Recommendation P.851 (2003) has been approved. It summarizes on the one hand the factors of the system, of the service and of the user which influence the service quality. On the other hand, guidelines are presented on how to evaluate services with the help of subjective interaction experiments, in order to determine the user’s quality perceptions. The relationships between influencing factors and perceived quality dimensions are displayed with the help of a taxonomy. This taxonomy puts different quality aspects into a logical relationship, and shows which factors have to be taken into account in the experimental set-up. The article discusses what has been reached in the new Recommendation, but also what is still missing in order to get more analytic information about the perform-ance of system characteristics and their influence on overall service quality.

Resources

Details