BERTScore

Automatic Evaluation Metric described in the paper BERTScore: Evaluating Text Generation with BERT (ICLR 2020). We now support about 130 models (see this spreadsheet for their correlations with human evaluation). Currently, the best model is microsoft/deberta-xlarge-mnli, please consider using it instead of the default roberta-large in order to have the best correlation with human evaluation.

News:

• Updated to version 0.3.13

  • Fix bug with transformers version > 4.17.0 (#148)

• Updated to version 0.3.12

  • Having get_idf_dict compatible with DDP (#140)
  • Fix setup bug (#138)

• Updated to version 0.3.11

  • Support 6 DeBERTa v3 models
  • Support 3 ByT5 models

• Updated to version 0.3.10

  • Support 8 SimCSE models
  • Fix the support of scibert (to be compatible with transformers >= 4.0.0)
  • Add scripts for reproducing some results in our paper (See this folder)
  • Support fast tokenizers in huggingface transformers with --use_fast_tokenizer. Notably, you will get different scores because of the difference in the tokenizer implementations (#106).
  • Fix non-zero recall problem for empty candidate strings (#107).
  • Add Turkish BERT Supoort (#108).

• Updated to version 0.3.9

  • Support 3 BigBird models
  • Fix bugs for mBART and T5
  • Support 4 mT5 models as requested (#93)

• Updated to version 0.3.8

  • Support 53 new pretrained models including BART, mBART, BORT, DeBERTa, T5, BERTweet, MPNet, ConvBERT, SqueezeBERT, SpanBERT, PEGASUS, Longformer, LED, Blendbot, etc. Among them, DeBERTa achives higher correlation with human scores than RoBERTa (our default) on WMT16 dataset. The correlations are presented in this Google sheet.
  • Please consider using --model_type microsoft/deberta-xlarge-mnli or --model_type microsoft/deberta-large-mnli (faster) if you want the scores to correlate better with human scores.
  • Add baseline files for DeBERTa models.
  • Add example code to generate baseline files (please see the details).

• Updated to version 0.3.7

  • Being compatible with Huggingface's transformers version >=4.0.0. Thanks to public contributers (#84, #85, #86).

• See #22 if you want to replicate our experiments on the COCO Captioning dataset.

• For people in China, downloading pre-trained weights can be very slow. We provide copies of a few models on Baidu Pan.

  • roberta-large password: dhe5
  • bert-base-chinese password: jvk7
  • bert-base-multilingual-cased password: yx3q

• Huggingface's datasets library includes BERTScore in their metric collection.

Previous updates

• Updated to version 0.3.6
  • Support custom baseline files #74
  • The option --rescale-with-baseline is changed to --rescale_with_baseline so that it is consistent with other options.
• Updated to version 0.3.5
  • Being compatible with Huggingface's transformers >=v3.0.0 and minor fixes (#58, #66, #68)
  • Several improvements related to efficency (#67, #69)
• Updated to version 0.3.4
  • Compatible with transformers v2.11.0 now (#58)
• Updated to version 0.3.3
  • Fixing the bug with empty strings issue #47.
  • Supporting 6 ELECTRA models and 24 smaller BERT models.
  • A new Google sheet for keeping the performance (i.e., pearson correlation with human judgment) of different models on WMT16 to-English.
  • Including the script for tuning the best number of layers of an English pre-trained model on WMT16 to-English data (See the details).
• Updated to version 0.3.2
  • Bug fixed: fixing the bug in v0.3.1 when having multiple reference sentences.
  • Supporting multiple reference sentences with our command line tool.
• Updated to version 0.3.1
  • A new BERTScorer object that caches the model to avoid re-loading it multiple times. Please see our jupyter notebook example for the usage.
  • Supporting multiple reference sentences for each example. The score function now can take a list of lists of strings as the references and return the score between the candidate sentence and its closest reference sentence.

Please see release logs for older updates.

Authors:

• Tianyi Zhang*
• Varsha Kishore*
• Felix Wu*
• Kilian Q. Weinberger
• Yoav Artzi

*: Equal Contribution

Overview

BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity. It has been shown to correlate with human judgment on sentence-level and system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks.

For an illustration, BERTScore recall can be computed as

If you find this repo useful, please cite:

@inproceedings{bert-score,
  title={BERTScore: Evaluating Text Generation with BERT},
  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
  booktitle={International Conference on Learning Representations},
  year={2020},
  url={https://openreview.net/forum?id=SkeHuCVFDr}
}

Installation

• Python version >= 3.6
• PyTorch version >= 1.0.0

Install from pypi with pip by

pip install bert-score

Install latest unstable version from the master branch on Github by:

pip install git+https://github.com/Tiiiger/bert_score

Install it from the source by:

git clone https://github.com/Tiiiger/bert_score
cd bert_score
pip install .

and you may test your installation by:

python -m unittest discover

Usage

Python Function

On a high level, we provide a python function bert_score.score and a python object bert_score.BERTScorer. The function provides all the supported features while the scorer object caches the BERT model to faciliate multiple evaluations. Check our demo to see how to use these two interfaces. Please refer to bert_score/score.py for implementation details.

Running BERTScore can be computationally intensive (because it uses BERT :p). Therefore, a GPU is usually necessary. If you don't have access to a GPU, you can try our demo on Google Colab

Command Line Interface (CLI)

We provide a command line interface (CLI) of BERTScore as well as a python module. For the CLI, you can use it as follows:

1. To evaluate English text files:

We provide example inputs under ./example.

bert-score -r example/refs.txt -c example/hyps.txt --lang en

You will get the following output at the end:

roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0) P: 0.957378 R: 0.961325 F1: 0.959333

where "roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)" is the hash code.

Starting from version 0.3.0, we support rescaling the scores with baseline scores

bert-score -r example/refs.txt -c example/hyps.txt --lang en --rescale_with_baseline

You will get:

roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)-rescaled P: 0.747044 R: 0.770484 F1: 0.759045

This makes the range of the scores larger and more human-readable. Please see this post for details.

When having multiple reference sentences, please use

bert-score -r example/refs.txt example/refs2.txt -c example/hyps.txt --lang en

where the -r argument supports an arbitrary number of reference files. Each reference file should have the same number of lines as your candidate/hypothesis file. The i-th line in each reference file corresponds to the i-th line in the candidate file.

2. To evaluate text files in other languages:

We currently support the 104 languages in multilingual BERT (full list).

Please specify the two-letter abbreviation of the language. For instance, using --lang zh for Chinese text.

See more options by bert-score -h.

3. To load your own custom model: Please specify the path to the model and the number of layers to use by --model and --num_layers.

bert-score -r example/refs.txt -c example/hyps.txt --model path_to_my_bert --num_layers 9

4. To visualize matching scores:

bert-score-show --lang en -r "There are two bananas on the table." -c "On the table are two apples." -f out.png

The figure will be saved to out.png.

5. If you see the following message while using BERTScore, please ignore it. This is expected.

Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Practical Tips

• Report the hash code (e.g., roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)-rescaled) in your paper so that people know what setting you use. This is inspired by sacreBLEU. Changes in huggingface's transformers version may also affect the score (See issue #46).
• Unlike BERT, RoBERTa uses GPT2-style tokenizer which creates addition " " tokens when there are multiple spaces appearing together. It is recommended to remove addition spaces by sent = re.sub(r' +', ' ', sent) or sent = re.sub(r'\s+', ' ', sent).
• Using inverse document frequency (idf) on the reference sentences to weigh word importance may correlate better with human judgment. However, when the set of reference sentences become too small, the idf score would become inaccurate/invalid. We now make it optional. To use idf, please set --idf when using the CLI tool or idf=True when calling bert_score.score function.
• When you are low on GPU memory, consider setting batch_size when calling bert_score.score function.
• To use a particular model please set -m MODEL_TYPE when using the CLI tool or model_type=MODEL_TYPE when calling bert_score.score function.
• We tune layer to use based on WMT16 metric evaluation dataset. You may use a different layer by setting -l LAYER or num_layers=LAYER. To tune the best layer for your custom model, please follow the instructions in tune_layers folder.
• Limitation: Because BERT, RoBERTa, and XLM with learned positional embeddings are pre-trained on sentences with max length 512, BERTScore is undefined between sentences longer than 510 (512 after adding [CLS] and [SEP] tokens). The sentences longer than this will be truncated. Please consider using XLNet which can support much longer inputs.

Default Behavior

Default Model

Language	Model
en	roberta-large
en-sci	allenai/scibert_scivocab_uncased
zh	bert-base-chinese
tr	dbmdz/bert-base-turkish-cased
others	bert-base-multilingual-cased

Default Layers

Please see this Google sheet for the supported models and their performance.

Acknowledgement

This repo wouldn't be possible without the awesome bert, fairseq, and transformers.

BERTScore для русского языка

Этот репозиторий является форком оригинальной реализации BERTScore, расширенной для оценки сгенерированных текстов на русском языке. Основная цель данной работы — исследование и выбор наиболее релевантных векторных представлений для текстов на русском языке в рамках метрики BERTScore. В результате исследования мы сравнили 30 моделей, поддерживающих русский язык. Наиболее релевантные векторные представления принадлежат 20 слою модели "ai-forever/ru-en-RoSBERTa".

Для более подробного изучения нашего исследования вы можете ознакомиться с презентацией.

Результаты:

Корреляции метрик для текстов, сгенерированных Gigachat

Dataset	Best embedding	BERTScore	Best embedding (ru)	BERTScore (ru)	BLEU	ROUGE-1	ROUGE-2	ROUGE-L
tg	microsoft/mdeberta-v3-base (7)	0.827	ai-forever/ruBert-base (22)	0.825	0.423	0.645	0.497	0.607
ru_simple_sent_eval	google/byt5-large (31)	0.615	ai-forever/ru-en-RoSBERTa (20)	0.673	0.096	0.281	0.149	0.25
science	facebook/mbart-large-50 (10)	0.749	ai-forever/ru-en-RoSBERTa (20)	0.749	0.282	0.599	0.481	0.560
dialogsum_ru	facebook/mbart-large-cc25 (11)	0.447	ai-forever/ru-en-RoSBERTa (7)	0.415	0.158	0.236	0.114	0.247
reviews_russian	facebook/mbart-large-50-many-to-many-mmt (6)	0.678	ai-forever/ru-en-RoSBERTa (20)	0.655	0.178	0.346	0.206	0.346
yandex	google/byt5-base (6)	0.433	ai-forever/ru-en-RoSBERTa (23)	0.454	0.078	0.192	0.165	0.192

AVG (слой)	google/byt5-large (29)	0.594	ai-forever/ru-en-RoSBERTa (20)	0.630	0.191	0.379	0.257	0.359
AVG (модель)	google/byt5-large	0.561	ai-forever/ruBert-base	0.564	0.191	0.379	0.257	0.359

Корреляции метрик для текстов, сгенерированных YandexGPT

Dataset	Best embedding	BERTScore	Best embedding (ru)	BERTScore (ru)	BLEU	ROUGE-1	ROUGE-2	ROUGE-L
tg	microsoft/mdeberta-v3-base (8)	0.299	ai-forever/ruBert-base (7)	0.345	0.157	0.238	0.197	0.239
ru_simple_sent_eval	xlm-roberta-large (16)	0.224	ai-forever/ru-en-RoSBERTa (21)	0.170	0.068	0.079	0.051	0.067
science	distilbert-base-multilingual-cased (5)	0.217	ai-forever/ru-en-RoSBERTa (20)	0.199	0.017	0.116	0.134	0.122
dialogsum_ru	google/mt5-xl (23)	0.291	ai-forever/ru-en-RoSBERTa (22)	0.333	0.077	0.149	0.157	0.155
reviews_russian	google/mt5-xl (23)	0.418	ai-forever/ruSciBERT (10)	0.397	0.138	0.234	0.176	0.213
yandex	google/byt5-base (10)	0.509	ai-forever/ruBert-base (0)	0.488	0.169	0.303	0.214	0.301

AVG (слой)	google/byt5-base (10)	0.284	ai-forever/ru-en-RoSBERTa (20)	0.275	0.096	0.171	0.138	0.166
AVG (модель)	facebook/mbart-large-50-many-to-many-mmt	0.249	ai-forever/ruBert-base	0.260	0.096	0.171	0.139	0.166

Корреляции BERTScore для наиболее релевантных векторных представлений по всем моделям

Модель	Кол-во параметров	Слой	Pearson
ai-forever/ru-en-RoSBERTa	404M	20	0.453
google/byt5-base	528M	8	0.447
google/byt5-large	1.23B	29	0.442
facebook/mbart-large-50-many-to-many-mmt	611M	10	0.433
facebook/mbart-large-50	611M	10	0.430
ai-forever/ruBert-large	427M	22	0.424
google/mt5-xl	3.7M	23	0.421
microsoft/mdeberta-v3-base	280M	8	0.421
ai-forever/ruBert-base	178M	10	0.421
google/mt5-large	1.2B	22	0.413
facebook/mbart-large-cc25	610M	9	0.410
xlm-mlm-100-1280	570M	15	0.403
bert-base-multilingual-cased	179M	6	0.401
ai-forever/FRED-T5-large	820M	13	0.401
ai-forever/ruRoberta-large	355M	20	0.399
bond005/rubert-entity-embedder	180M	4	0.398
DeepPavlov/rubert-base-cased	180M	7	0.398
xlm-roberta-base	279M	6	0.397
xlm-roberta-large	561M	16	0.389
cointegrated/rubert-tiny2	29М	2	0.385
kazzand/ru-longformer-tiny-16384	34.5М	2	0.379
distilbert-base-multilingual-cased	135M	3	0.376
ai-forever/ruSciBERT	123M	10	0.376
kazzand/ru-longformer-large-4096	434М	6	0.373
google/mt5-small	300M	3	0.371
google/mt5-base	580M	4	0.365
ai-forever/ruT5-base	222M	0	0.358
google/byt5-small	300M	1	0.341
ai-forever/ruT5-large	737M	0	0.341
kazzand/ru-longformer-base-4096	148М	6	0.327