Request Correction

JSTS-Neg: Japanese Semantic Textual Similarity Dataset for Evaluating Negation Understanding Ability

Negation is a common linguistic phenomenon in natural language. Thus, datasets and benchmarks focused on negation are being constructed to evaluate the negation understanding abilities of language models. Negation is especially crucial when estimating the semantic similarity between sentences because it inverses their meaning. Although semantic textual similarity (STS) is one of the useful tasks to evaluate the abilities of large language models (LLMs), few STS datasets focus on negation. In this research, we introduce JSTS-Neg, a new Japanese STS dataset focusing on negation. Most instances in JSTS-Neg include negations and they are composed of both clausal and sub-clausal negations to reflect a variety of negation types. Moreover, JSTS-Neg consists of negation minimal pairs that only differ in the presence or absence of a negation cue. We evaluate the performance of existing LLMs on JSTS-Neg using negation minimal pairs to explore their abilities and limitations in understanding negation. LLMs tend to predict the similarity of two sentences ignoring negation cues in specific settings.