Evaluating Style Embeddings for Machine-Generated Text Detection

Abstract

In this paper, we evaluate the use of style embeddings for distinguishing machine-generated from human-written text. Style embeddings are particularly suited for this task as compared to semantic embeddings, they offer higher content-independence, and compared to feature-engineering approaches, they offer a richer and more holistic representation of writing style. We use a detection module in which texts are first embedded in high-dimensional stylistic spaces using a style encoder, and the resulting vector representations are classified using supervised methods. To optimize this detector, we evaluate the performance of a range of pre-trained public-domain style encoders paired with different supervised methods. When evaluated on MGTBench, a widely adopted benchmark, our approach matches or exceeds state-of-the-art performance metrics. It also generalizes well across various text domains and LLMs. Our findings highlight the potential, and would facilitate the use, of style embeddings as lightweight and effective components of machine-generated text detection systems.

Resources

Details