Cultural Grounding in Swedish: Extending an Everyday Knowledge Benchmark for LLMs

Abstract

Benchmarks for evaluating Large Language Models (LLMs) on everyday knowledge across cultures and languages are increasingly used to assess cultural competence and contextual understanding. However, many multilingual extensions rely primarily on translated question–answer pairs, limiting their ability to capture locally grounded variation. In this work, we present a Swedish extension of an existing cross-cultural everyday knowledge benchmark in which questions are translated into Swedish, and answers are individually collected from five participants, coming from diverse social and professional backgrounds. This design enables us to capture culturally situated, naturally produced responses rather than transferred or translated answer templates. We document the translation protocol, annotators, and agreement analysis, and examine variation across annotators as a signal of culturally contingent knowledge. We evaluate several state-of-the-art multilingual and instruction-tuned LLMs against the aggregated human responses and analyze model performance. Our results reveal that while models often approximate prototypical answers, they struggle with culturally specific nuances and intra-cultural variation. The Swedish extension provides a resource for studying culturally grounded evaluation and highlights the importance of human-generated local answers when benchmarking LLMs across languages.

Resources

Details