Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining

Abstract

In this research, we introduce LoRA probing, a lightweight approach for observing how core syntactic abilities emerge during LLM pretraining. Leveraging OLMo‑2’s public intermediate checkpoints, we trace learning curves across 24 pretraining stages on 33 Universal Dependencies languages by fine‑tuning LoRA with step-by-step parsing instructions and a simple tabular output. To fit the relatively short context length of the OLMo-2, we design a compact 2‑step‑no‑form prompt template and this matches the baseline in average accuracy while halving the context length and substantially increasing throughput, enabling efficient large‑scale evaluation. Token Recall surpasses 0.9 within the first 1–2K pretraining steps, indicating that stable output formatting emerges early. Despite OLMo‑2‑7B’s English‑centric pretraining, LAS exceeds 80 points in 29 of 33 languages; however, relations such as iobj and csubj show delayed onset and instability across many languages. LoRA probing thus provides a practical, reproducible lens on the cross‑lingual dynamics of syntactic acquisition during LLM pretraining.

Resources

Details