Designing LLM Agents for User-Centered Language Service Selection

Abstract

With the rapid expansion of language resources and services across repositories and platforms, users face an overwhelming number of options. While this diversity promises flexibility, non-experts struggle to compose appropriate resource pipelines and select services that satisfy both functional and non-functional requirements. We propose a user-centered framework of LLM agents that interprets natural-language requests and performs end-to-end language service selection. The agents extract functional requirements to form coherent task compositions and select suitable language services for each component by interpreting non-functional quality aspects embedded in contextual cues. To ensure reliable and explainable decisions, we employ a four-step structured reasoning procedure that combines Few-Shot exemplars and Chain-of-Thought reasoning: extracting functional requirements, inducing non-functional evaluation axes, applying these axes as constraints in candidate retrieval, and determining a final composition. We construct a benchmark dataset pairing diverse user requests with standardized language service profiles containing metadata and quality indicators, and evaluate our framework against representative prompting-based baselines. Results show consistent gains in Precision, Recall, and F1-score, demonstrating improved capture of both functional intent and quality preferences. These findings demonstrate that structured LLM agents can bridge natural-language user intents and language service configurations, enabling end-to-end selection and composition in a transparent and user-centered manner.

Resources

Details