LLM Probe: Evaluating LLMs for Low-Resource Languages

Abstract

Despite the rapid progress of large language models (LLMs), their linguistic capabilities in low-resource and morphologically rich languages remain insufficiently understood due to the scarcity of annotated resources and the lack of standardised evaluation frameworks. This paper introduces LLM Probe, a lexicon-based evaluation framework for systematically assessing the linguistic competence of LLMs in low-resource language settings. The framework evaluates models across four dimensions of language understanding: lexical alignment, part-of-speech identification, morphosyntactic probing, and translation fidelity. To demonstrate the framework, we construct a manually annotated benchmark dataset using a low-resource Semitic language as a case study. The dataset consists of bilingual lexicons enriched with linguistic annotations, including part-of-speech categories, grammatical gender, and morphosyntactic features, with high inter-annotator agreement ensuring annotation reliability. We evaluate a diverse set of models spanning causal language models and sequence-to-sequence architectures. The results reveal substantial variation in model performance across linguistic tasks: sequence-to-sequence models generally achieve stronger performance in morphosyntactic analysis and translation quality, while causal models exhibit competitive lexical alignment but weaker translation fidelity. Our findings highlight the importance of linguistically grounded evaluation for understanding the limitations of LLMs in low-resource contexts. We release LLM Probe and the accompanying benchmark dataset as open-source resources to support reproducible benchmarking and to advance the development of more inclusive multilingual language technologies.

Resources

Details