Sensitivity of IR systems Evaluation to Topic Difficulty

Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002)

Abstract

The difficulty of the topics or queries is one of important factors in evaluating information retrieval (IR) systems. This paper analyzes the differences of system ranking affected by the topic difficulty using a test collection ’NTCIR-1,’ which is constructed for evaluating Japanese IR systems and composed of (1) the topics, (2) the document database, and (3) the lists of relevant judgments. Furthermore, this paper defines measures for the various features on the topics, and analyzes the correlation between them, in order to investigate the predictability of the topic difficulty.