Beyond PARSEVAL
Towards Improved Evaluation Measures for Parsing Systems

Schedule (Final Version)

9:15 – 9:30 Opening: Aims of the workshop

9:30 – 10:00 Relational evaluation schemes
Ted Briscoe, John Carroll, Jonny Graham, Ann Copestake
Universities of Cambridge & Sussex

10:00 – 10:30 Towards a dependency-oriented evaluation for partial parsing
Sandra Kübler, Heike Telljohann
University of Tübingen

10:30 – 11:00 LinGO Redwoods -- a rich and dynamic treebank for HPSG
Stephan Oepen, Ezra Callahan, Dan Flickinger, Christoper D. Manning
Stanford University


Coffee Break (30 mins)

11:30 – 12:30 Panel: Parser evaluation in context
John Carroll, Patrick Paroubek, Owen Rambow, Hans Uszkoreit


Lunch Break

14:00 – 14:30 A test of the leaf-ancestor metric for parse accuracy
Geoffrey Sampson, Anna Babarczy
University of Sussex

14:30 – 15:00 Evaluating parser accuracy using edit distance
Brian Roark
AT&T-Labs Research


Short Break (10 mins)

15:10 – 15:40 Evaluating syllabification: one category shared by many grammars
Karin Müller
Saarland University

15:40 – 16:10 Towards comparing parsers from different linguistic frameworks:
an information theoretic approach
Gabriele Musillo, Khalil Sima'an
University of Amsterdam

16:10 – 16:40 Evaluation of the Gramotron parser for German
Franz Beil, Detlef Prescher, Helmut Schmid, Sabine Schulte im Walde
TEMIS Paris, DFKI Saarbrücken & IMS Stuttgart


Coffee Break (30 mins)

17:10 – 17:40 Evaluating a wide-coverage CCG parser
Stephen Clark, Julia Hockenmaier
University of Edinburgh

17:40 – 18:10 A comparison of evaluation metrics for a broad coverage parser
Richard Crouch, Ronald M. Kaplan, Tracy H. King, Stefan Riezler
Palo Alto Research Center

18:10 – 20:00 Wrap up and kick-off: Initiatives and action plans (open end)