PLABA-EVAL: A Multi-Dimensional, In-Context Sentence Readability Dataset for Medical Text
Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026
Abstract
We present an in-context framework for assessing readability that separates reading difficulty into multiple subjective dimensions. Participants read biomedical abstracts with full-document access and provide sentence-level ratings of Processing Ease and Perceived Understanding, followed by an open-book multiple-choice comprehension check. Using this protocol, we release PLABA-EVAL, a dataset of 78 biomedical abstracts and expert plain-language adaptations (609 sentences), annotated by three independent raters per document. Analyses show that Ease and Understanding are strongly related but not interchangeable, and that perceived understanding aligns more closely with open-book comprehension performance. We provide baseline linguistic analyses for both dimensions, illustrating how the dataset supports work on readability, simplification, and sentence-level difficulty modeling.