When Neutral Turns Negative: Cross-Domain Failure Modes in Hinglish Political Sentiment Analysis

Abstract

Sentiment analysis models are increasingly deployed to analyze political discourse, yet strong in-domain performance does not guarantee robustness under domain shift. We study cross-domain generalization in Hinglish (Hindi–English code-mixed) sentiment analysis by evaluating a fine-tuned XLM-RoBERTa classifier, trained on 29,000 general-domain Hinglish sentences, on a curated benchmark of politically oriented Hinglish text. While the model achieves 92.02% accuracy in-domain, performance drops to 71.83% under political domain shift. Error analysis reveals a pronounced directional bias with 48.9% of neutral political statements misclassified as negative, indicating a systematic neutrality-to-negative shift. In addition, 87.5% of incorrect predictions are assigned confidence scores above 95%, pointing to severe miscalibration under distribution shift. We further compare these results against an instruction-tuned large language model (Llama 3.3), which achieves 90.85% zero-shot accuracy and 94.37% accuracy with contextual prompting, while substantially reducing neutrality bias. Our findings indicate the need for domain-aware evaluation, calibration diagnostics, and explicit reporting of failure modes when deploying sentiment models in politically sensitive settings.

Resources

Details