Add metrics to compute fluency of references

[poojah15]

Apart from the content preservation metrics that this repo contains, it is also common to check the fluency/ coherence of the generated outputs. Forward PPL using a pretrained language model is generally used to measure the fluency of the outputs but there seem to be other newly introduced metrics as well... I was wondering if anyone knows of some tools to compute fluency of sentences apart from PPL? It would be a useful feature to add to this repository as well. Any help is much appreciated!

[sharan21]

Hey, you can check out the "Naturalness" metric as a substitute for perplexity as it correlates better with human judgment in measuring the fluency or coherence of a sentence. https://github.com/passeul/style-transfer-model-evaluation
Ill check it out myself and add this feature as I feel it will be useful as well!

[poojah15]

Thanks! It seems to be just the tool I need. Yes, it will be useful to integrate it as a feature to this repo.

[sharan21]

I tested the naturalness metric in https://github.com/passeul/style-transfer-model-evaluation and it works quite nicely. It should be useful in Text Style Transfer and Summarization tasks or even just Generation tasks to check the quality. Integrating it into this repo should be straightforward as well. Ill link a PR to this issue and merge it for others to use :)

[sharan21]

I have tested the new feature on my branch and it is working. This is the pull request for the issue. Feel free to clone my branch and use it incase it takes time for the pull request to get merged.

[poojah15]

Thanks a lot!

[temporaer]

@poojah15 , were you able to successfully run from @sharan21 's code?