A Measure of Systematic Disagreement

Abstract

We introduce a new metric that quantifies the extent of systematicity of the disagreement between annotators. The metric, called σ, is inspired by Structural Balance Theory and it approximates the clusterability of the annotators of a dataset. Paired with a standard metric of inter-annotator agreement such as Krippendorffs α, σ measures the amount of disagreement which stems from genuine subjective factors as opposed to the amount of disagreement caused by inner features of the annotation task. The metric is applied to over twenty datasets encoding a broad variety of annotations, showing its effectiveness in capturing the systematicity of annotator disagreement and its explanatory value.

Resources

Details