HumaniCA: A Benchmark Resource for the Detection of Users' Ascription of Humanness to Conversational Agents

Abstract

Anthropomorphizing, which involves attributing human-like characteristics to non-human entities, is common in users’ conversations with text-based conversational agents and can lead to a misalignment between the users’ expectations and the agent’s actual capabilities. Detecting users’ ascriptions of humanness automatically may enable systems to identify when users adopt a human-like style when conversing with an agent and to adapt its responses accordingly to tune their expectations. In this paper, we introduce HumaniCA, a benchmark resource comprising three annotated datasets of user turns from real dialogues with three different types of conversational agents (task-oriented, Q&A, and LLM-based) aimed at indicating whether the user is ascribing humanness to the conversational agent. We also identified a set of linguistic indicators of user ascription of humanness to conversational agents and validated their utility with benchmark experiments. We then compared performance of our linguistic features and other well-known textual features (TF-IDF weights and SentenceBERT word embeddings), as well as their combinations. The evaluation highlights the central role of our linguistic features: whether used individually or in combination, they consistently achieve higher accuracy across all agent types.

Resources

Details