ResQu: A Framework for Automatic Evaluation of Knowledge-Driven Automatic Summarization

TitleResQu: A Framework for Automatic Evaluation of Knowledge-Driven Automatic Summarization
Publication TypeThesis
Year of Publication2016
AuthorsNishita Jaykumar
Academic DepartmentDepartment of Engineering & Computer Science
Number of Pages64
Date Published05/2016
UniversityWright State University
KeywordsAbstractive Summaries, Automatic Summarization., Summarization Evaluation

Automatic generation of summaries that capture the salient aspects of a search resultset (i.e., automatic summarization) has become an important task in biomedical research. Automatic summarization offers an avenue for overcoming the information overload problem prevalent in large online digital libraries. However, across many of the knowledge-driven approaches for automatic summarization it is not always clear which features highly impact or influence the quality of a summary. Instead, there has been considerable focus on utilizing schema knowledge to facilitate browsing and exploration of generated summaries a posteriori. Such informative features should not be ignored, since they could be utilized to help optimize the models that generate these semantic summaries in the first place. In this research, we adopt a leave-one-out approach to assess the impact of various features on the quality of automatically generated summaries that contain structured background knowledge. We first create the gold standard summaries, using information-theoretic methods, by extraction and validation, then the semantic summaries are transformed into an equivalent textual format. Finally, various similarity metrics, such as cosine similarity, euclidean distance, and Jensen-Shannon divergence are computed under different feature combinations, to assess summary quality against the textual gold standard. We report on the relative importance of the various features used to automatically generate the semantic summaries in a biomedical application. Our evaluation suggests that the proposed approach is an effective automatic evaluation method for assessing feature importance in automatically generated semantic summaries.

Full Text