Code-switching as a Bias Indicator in LLMs: "the Consequences Are Not the Same Para Nosotros"

Abstract

Code-switching is a widespread linguistic practice among bilingual speakers. While recent studies have addressed the impact of code-switching on downstream task performance, the potential biases and harms that language models may cause when prompted with code-switching have yet to be investigated. The objective of this study is to investigate whether code-switching constitutes an implicit indicator of ethnicity that can be leveraged to unveil covert racist or xenophobic bias in language models. The present paper introduces a methodology to compare generated texts that were prompted with code-switching vs. with monolingual inputs. It is applied on both Hinglish and Spanglish, two popular forms of code-switching that are omnipresent in Indian and Hispanic communities. With a decision tree approach, we tackle various types of semantic differences through the use of semantic resources, stereotypes lists, POS-tagging and sentiment classifiers. Over 84k text pairs are generated with 3 popular large language models. Overall, around 50% of generated text pairs are not semantically equivalent, and 25% of the time, there is a potential for harm against the Indian or Hispanic community. The different possible harms are further discussed, relying on sociological studies to argue that bias and harms against socially discriminated communities have greater consequences.

Resources

Details