MOTIVATION
Individual organizations and countries have been investing in the creation of resources and methods to support the advancement of multimodal technologies, products and applications. This is evident in US DARPA and ARDA HLT programs, the EU HLT programme under FP5-IST, the German MTI Program, the Francophone AUF programme and others. The European 6th Framework program (FP6 ), planned for a start in 2003, includes multilingual and multisensorial communication as major R&D issues. Substantial mutual benefits can be expected from addressing these issues through international cooperation. Nowhere is this more important than in the relatively new areas of multimedia (i.e., text, audio, video), multimodal (visual, auditory, tactile), and multicodal (language, graphics, gesture) communication.
OBJECTIVE
The primary purpose of this one day workshop (feeding into a subsequent half day Multimodal Roadmap workshop) is to report and discuss multimodal resources, annotation standards, tools and methods, and evaluation metrics/methods, as well as strategize jointly about the way forward. The workshop consists of short presentations and facilitated sessions with the intent of jointly identifying grand challenge problems, a shared understanding of and plan for multimedia resources and applications, and identification of methods for facilitating the creation of multi-media resources.
PROGRAM
8:00 a.m. Welcome
Mark Maybury (MITRE, USA) and Jean-Claude Martin (LIMSI-CNRS, France)
Resources and Annotation: Multimodal
8:30 a.m. Data Resources and Annotation Schemes for Natural Interactivity
Laila Dybkjær and Niels Ole Bernsen (University of Southern Denmark, Denmark)
8:50 a.m. Metadata Set and Tools for Multimedia/Multimodal Language Resources
P. Wittenburg, D. Broeder, Freddy Offenga, and Don Willems (Max Planck Institute for Psycholinguistics, The Netherlands)
Resources and Annotation: Gesture and Speech
9:10 a.m. FORM: A Kinematic Annotation Scheme and Tool for Gesture Annotation
Craig Martell, Chris Osborn, Jesse Friedman and Paul Howard (University of Pennsylvania, USA)
9:30 a.m. Cross-Linguistic Studies of Multimodal Communication
P. Wittenburg, S. Kita, and H. Brugman
(Max Planck Institute for Psycholinguistics, The Netherlands)
Resources and Annotation: Facial Expressions, Speech, Integration
9:50 a.m. Development of User-State Conventions for
Multimodal Corpus in SmartKom
Silke Steininger, Susen Rabold, Olga Dioubina, and Florian Schiel
(Ludwig-Maximilians University, Munich, Germany)
10:10 a.m. Integration of multi-modal data and annotations into a simple
extendable form: the extension of the BAS Partitur Format
Florian Schiel, Silke Steininger, Nicole Beringer,
Ulrich Tuerk, and Susen Rabold (University of Munich, Germany)
10:40 am Multimodal Resources Group Discussion
All
11:00 11:20 a.m. Morning Break
Annotation Tools
11:20 am Multimodal Corpus Authoring System
Anthony Baldry (Univ. of Pavia, Italy) and Christopher Taylor (Univ. of
Trieste, Italy)
11:40 am The Observer Video-Pro: Professional system for collection,
analysis and presentation of observational data
Niels Cadée, Erik Meyer, Hans Theuws, and Lucas Noldus
(Noldus Information Technology, The Netherlands)
13:00 p.m. Lunch
Multimodal Fusion
14:30 p.m. Prosody based co-analysis of Deictic Gestures and Speech
in Weather Narration Broadcast
Kettebekov Sanshzar,Yeasin Mohammed, Krahnstoever Nils, SharmaRajeev
(Dept. of CS and Engineering, Pennsylvania State University, USA)
14:50 p.m. A Generic Formal Description Technique for
Fusion Mechanisms of Multimodal Interactive Systems
Philippe Palanque and Amélie Schyn
(LIIHS IRIT, Université Toulouse, France)
Research Infrastructure
15:10 p.m. Eye Bed
Ted Selker, Winslow Burleson, Jessica Scott, and Mike Li
(MIT Media Lab, Cambridge, USA)
15:40 pm. MUMIN: A Nordic Network for MUltiModal Interfaces
Patrizia Paggio (Center for Sprogteknologi, Copenhagen),
Kristiina Jokinen (University of Art and Design, Helsinki), and
Arne Jönsson (University of Linköping)
System Evaluation
16:00 pm PROMISE - A Procedure for Multimodal Interactive
System Evaluation
Nicole Beringer, Ute Kartal, Katerina Louka, Florian Schiel, Uli Türk
(University of Munich, Germany)
16:30 - 17:00 p.m. Afternoon Break
17:00 p.m. Final Group Discussion
All
18:00 p.m. Close
KEY REFERENCES
* LREC2000 multimodal corpora workshop
(www.mpi.nl/world/ISLE/events/LREC%202000/LREC2000.htm)
* Working group on Natural Interaction and Multimodality (NIMM) under
the IST-ISLE project (http://isle.nis.sdu.dk,
(www.ldc.upenn.edu/sb/isle.html).
* ELRA Multimodal corpora survey (http://www.icp.inpg.fr/ELRA/).
* Dagstuhl Seminar on Multimodal Fusion and Coordination questionnaire
on multimodal corpora
(www.dfki.de/~wahlster/Dagstuhl_Multi_Modality/)
* NIST Automatic Meeting Transcription Project
(http://www.nist.gov/speech/test_beds/mr_proj): "
* ATLAS (Architecture and Tools for Linguistic Analysis Systems)
(www.nist.gov/speech/atlas):
* NSF TALKBANK (www.talkbank.org)
REGISTRATION
The registration fee for the workshop is 90 EURO for conference
participants and 140 EURO for others. The fee includes two
coffee breaks and the proceedings of the workshop.
ORGANIZERS
Mark Maybury, The MITRE Corporation, Bedford, MA USA
maybury@mitre.org (co-chair)
Jean-Claude Martin, LIMSI-CNRS, LINC-University Paris 8, Orsay, France
martin@limsi.fr (co-chair)
PROGRAM COMMITTEE
Lisa Harper, The MITRE Corporation, USA lisah@mitre.org
Catherine Pelachaud, University of Rome "La Sapienza", Italy,
cath@dis.uniroma1.it
Michael Kipp, DFKI, Germany, kipp@dfki.de
Wolfgang Wahlster, DFKI, Germany, wahlster@dfki.uni-sb.de
Oliviero Stock, IRST, stock@irst.itc.it
Harry Bunt, Tilburg University, Harry.Bunt@kub.nl
Antonio Zampolli, Consiglio Nazionale delle Ricerche,pisa@ilc.pi.cnr.it
Steven Krauwer, ELSNET, steven.krauwer@elsnet.org
Bernsen Niels Ole, NISLab, University of Southern Denmark - Odense,
Denmark, nob@nis.sdu.dk
Dybkjaer Laila, NISLab, University of Southern Denmark - Odense,
Denmark, laila@nis.sdu.dk