Evaluation of Linguistics-Based Translation

Abstract

We report on the evaluation of the Norwegian-English MT prototype system LOGON. The system is rule-based and makes use of well-established frameworks for analysis and generation (LFG and HPSG). Minimal Recursion Semantics is the “glue” which performs transfer from source to target language and serves as the information vehicle between LFG and HPSG. The project-internal testing uses material from the training data sources from the domain guidebooks for mountain hiking in the summer season in Southern Norway. This testing, involving eight external assessors, yielded 57 % translated sentences, with acceptable fidelity measures, but with less than acceptable fluency measures. Additional test 1: The LOGON system is sensitive to vocabulary, so we were interested to see to what extent the system would be able to carry over to new texts from the same narrow domain. With only 22 % acceptable translations, this test had disappointing results. Additional test 2: Given the grammatical backbone of the system, we found it important to test it on a syntactic test-suite with only known vocabulary. Here, 55 % of the sentences had good translations. The tests show that even within a very narrow semantic domain, vocabulary sensitivity is the most crucial obstacle for this approach.

Resources

Details