Less Is More? The Role of Demographic Author Information in Emotion Classification of Ambiguous Text

Abstract

Emotion annotation in text is a challenging task that often yields low inter-annotator agreement. Missing context, differences in world knowledge and extra-linguistic factors such as the author’s identity influence how emotions are perceived. When the text does not provide sufficient information, details about the author may help resolve ambiguity. We test the hypothesis that providing annotators with demographic information reduces disagreement in emotion annotation. We compare one group of annotators who sees each text alongside demographic information about its author, with a group who sees only the text. We find in our study with 500 annotators and 250 texts that displaying demographic information about the author of the text does not improve agreement between annotators, nor does it improve agreement with the gold label. The only exception are cases where the emotion polarity (positive or negative) is unclear. We also find that annotators perform overall better at identifying the correct emotion label when it aligns with gender stereotypes. Zero-shot prompting experiments with large language models do resemble the human annotation experimental results. Our findings suggest that providing demographic information is not a straightforward remedy for ambiguity in emotion annotation and careful consideration is needed when incorporating such data.

Resources

Details