Skip to content

Latest commit

 

History

History
75 lines (56 loc) · 4.32 KB

README.md

File metadata and controls

75 lines (56 loc) · 4.32 KB
Raw

PragmEval : A Pragmatics-Centered Evaluation Framework for Natural Language Understanding

This is a data/code release accompanying this paper:

  • Title: "A Pragmatics-Centered Evaluation Framework for Natural Language Understanding"
  • Authors: Damien Sileo, Tim Van de Cruys, Camille Pradel and Philippe Muller
  • Accepted at LREC2022
  • https://arxiv.org/abs/1907.08672

Contents

PragmEval is a compilation of 11 evaluation datasets with a focus on discourse, that can be used for evaluation of English Natural Language Understanding, or as auxiliary training tasks for NLP models.

While the idea of meaning as use permeates NLP, it's not clear that current evaluations fully account for that aspect. Previous evaluation frameworks have no clear way to evaluate how models deal with implicatures or different kinds of speech acts, and are arguably focussing on semantics (Natural Language Inference of Semantic Similarity) rather than use. We propose a discourse-centered evaluation with a focus on meaning as use.

dataset categories example class #train
PDTB discourse relation it was censorship / it was outrageous conjunction 13k
STAC discourse relation what ? / i literally lost question-answer-pair 11k
GUM discourse relation do not drink / if underage in your country condition 2k
Emergent stance a meteorite landed in nicaragua. / small meteorite hits managua for 2k
SarcasmV2 presence of sarcasm don't quit your day job / [...] i was going to sell this joke. [...] sarcasm 9k
SwitchBoard speech act well , a little different , actually , hedge 19k
MRDA speect act yeah that 's that 's that 's what i meant . acknowledge-answer 14k
Verifiability verifiability I've been a physician for 20 years. verifiable-experiential 6k
Persuasion C/E/P/S/S/R Co-operation is essential for team work / lions hunt in a team low specificity 0.6k
Squinky I/I/F boo ya. uninformative, high implicature, unformal, 4k
EmoBank V/A/D I wanted to be there.. low valence, high arousal, low dominance 5k

Instructions

Recommended usage

from datasets import load_dataset
dataset = load_dataset('pragmeval','gum')

Evaluate your model with this script: https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing

Building from sources:

git clone https://github.com/synapse-developpement/PragmEval.git

The preprocessed datasets are available in the pragmeval folder in tsv format.

Run the bash get_data.bash in data to download dataset from the sources

Run the notebook Make PragmEval 1.0 after having specified pragmeval_base_path in the third cell to perform preprocessing and exports.

Citation

@inproceedings{sileo-etal-2022-pragmatics,
    title = "A Pragmatics-Centered Evaluation Framework for Natural Language Understanding",
    author = "Sileo, Damien  and
      Muller, Philippe  and
      Van de Cruys, Tim  and
      Pradel, Camille",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.255",
    pages = "2382--2394",
}

Contact

For further information, you can contact:

damien dot sileo at gmail dot com