Skip to content

Latest commit

 

History

History
88 lines (65 loc) · 3.32 KB

README.md

File metadata and controls

88 lines (65 loc) · 3.32 KB

ROSE 🌹

This repo is for our ACL 2023 paper "Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation". We provide the scripts for our RoSE benchmark and meta-evaluation.

Please visit here for a demo page of this project.

Quick Links

RoSE 🌹 Benchmark

RoSE can be downloaded with Hugging Face Datasets under Salesforce/rose. We provide a notebook, demo.ipynb, for basic usage of our dataset.

ACU Annotations

RoSE benchmark contains system outputs annotated with our ACU protocol. It contains four parts:

  • CNNDM, test set annotations
  • CNNDM, validation set annotations
  • XSum, test set annotations
  • SamSum, test set annotations

We summarize the statistics below.

Dataset Split #Doc. #Sys. #Total Summ. HF Name
CNNDM Test 500 12 6000 cnndm_test
CNNDM Validation 1000 8 8000 cnndm_validation
XSum Test 500 8 4000 xsum
SamSum Test 500 8 4000 samsum

Human Annotations with Different Evaluation Protocols

We have system outputs annotated with four different human evaluation protocols in total. We summarize them below.

Protocol w/ Input Document w/ Reference Summary Fine-grained
Prior ✗ ✗ ✗
Ref-free ✓ ✗ ✗
Ref-based ✗ ✓ ✗
ACU ✗ ✓ ✓

We annotated two sets of system summaries.

  1. Summaries of 12 fine-tuned systems. The huggingface data split name is cnndm_protocol.
  2. Zero-shot summaries from large language models (GPT3, T0), together with summaries from BRIO and BART. The huggingface data split name is cnndm_protocol_gpt3.

Meta-Evaluation

We provide scripts for statistical analysis of the meta-evaluation results in our paper.

correlation.py

functions for computing correlation coefficients

stat_test.py

functions for conducting statistical tests, including bootstrap, permutation test and computing confidence interval

power_analysis.py

functions for computing power analysis, please note that computing power analysis can be time-consuming, and please maximize the number of processes to speed up the computation

demo.py

demo script for utilizing the functions in the above files

Citation

Please cite our paper if you use RoSE in your work:

@inproceedings{Liu2022RevisitingTG,
  title={Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation},
  author={Yixin Liu and Alexander R. Fabbri and Pengfei Liu and Yilun Zhao and Linyong Nan and Ruilin Han and Simeng Han and Shafiq R. Joty and Chien-Sheng Wu and Caiming Xiong and Dragomir R. Radev},
  booktitle={Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics},
  year={2023},
}