Title

End-to-End Evaluation of Multimodal Dialogue Systems - can we Transfer Established Methods?

Authors

Nicole Beringer (Institut für Phonetik und Sprachliche Kommunikation, Schellingstr. 3, D-80799 München, Germany)

Katerina Louka (Institut für Phonetik und Sprachliche Kommunikation, Schellingstr. 3, D-80799 München, Germany)

Victoria Penide-Lopez (BIOMAX Informatics AG.Lochhamer Str. 11, D-82152 Martinsried, Germany )

Uli Türk (Institut für Phonetik und Sprachliche Kommunikation, Schellingstr. 3, D-80799 München, Germany)

Session

EP1: Evaluation

Abstract

The goal of this paper is to define a methodology for the end-to-end evaluation of the multimodal dialogue system SmartKom along the lines of the DARPA guidelines for spoken dialogue systems. The methodology consists of an extended  framework for the evaluation of a multimodal dialogue system, evaluation metrics for its various components,and an approach to compare the user satisfaction with the system's technical performance.

Keywords

Multimodality, SmartKom, Dialogue system evaluation, Evaluation methods

Full Paper

50.pdf