Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.

Abstract

Textual data analysis lies at the heart of inductive reasoning in corpus linguistics. Corpus-driven approaches place the corpus at the center of working hypotheses and use statistical processing as an exploratory tool. With deep neural networks, the training corpus is also crucial, but the objectives are less exploratory. Nevertheless, the performance of Transformers in automatic language processing suggests that self-attention is an effective means of extracting structural information from corpora. In this article, we present interdisciplinary work that uses Transformers descriptively to shed light on linguistic phenomena present in a learning corpus. We propose using two feature-based interpretation methods in a case study of political speeches applied to a text generation task. The first method is a global approach that uses attention scores to analyse the training corpus. The second is a local approach that uses gradient-based features to analyse predictions. These methods are compared to standard statistical techniques, providing empirical confirmation of the observed phenomena. We conclude on the potential of Transformers as a heuristic tool for corpus linguistics.

Resources

Details