Towards end-to-end reference-free summarization evaluation via negative sampling

Abstract

Evaluating machine-generated summaries without a human-written reference summary has been a need for a long time. In this paper, we present a proof-of-concept study to a summary evaluation approach without the presence of reference summaries. By negative sampling, massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. Two learning schemes are explored: weakly supervised learning with explicit number labels and preference learning with inexplicit labels. Extensive experiments on several datasets show that our approaches can produce scores highly correlated with human ratings.