Errors after installing via pip and fixes

[ankitk2109]

Hi Team,

I installed the summ_eval package using pip and also tried to run it locally by cloning it. I encountered several issues and here are the fixes that helped me get this running. I am putting them here for anyone who might face a similar issue or if the team is interested to update them directly into the code.

It would be a great help if these issues could be addressed directly inside the package. I would love to pull this work into my current project :)

Setting-up Rouge Metric inside summ_eval:

• If you have installed summ_eval using pip. You might have gotten errors while loading the RougeMetric() asking to do pip install -U git+https://github.com/bheinzerling/pyrouge.git again and again.

• On the windows machine, we have to change a few commands. I would recommend the pythonic way as it is platform-independent. We need to change the following lines after the line subprocess.run(["tar", "-xzf", "project.tar.gz"]) :

  shutil.move("Yale-LILY-SummEval-7e4330d/evaluation/summ_eval/ROUGE-1.5.5/", ROUGE_HOME)
  os.remove("project.tar.gz")
  shutil.rmtree("Yale-LILY-SummEval-7e4330d/")
  

Installation & Modifications:

• Missing Import
  • File ..\summ_eval\summa_qa_metric.py: Add from sys import stderr
• While evaluating super_metric I encountered an error in file utils.py. The error says {ModuleNotFound}No module named 'sentence_transformers' this occurs because it is trying to import a package named as sentence_transformers which is not present in the packages, however, that is inside the summ_eval package hence we want to import the relative package.
  • I updated the line from importlib.import_module(module_path) to importlib.import_module(f"{__name__.split('.')[0]}.{module_path}") which is results inimportlib.import_module('summ_eval.sentence_transformers.models') .
  • import_module can take a relative path if we mention the source package. You can check out this link https://stackoverflow.com/a/10675081 for more info.
• Missing requirements:
  • scikit-learn==0.23.2 , I was getting an error for sklearn since it was not installed. Not sure about the exact version by I installed 0.23.2

If setting up by cloning the repository (On Windows)

Pre-requisite:

• On windows machine:

  • Perl should be installed

  • To check if it is installed correctly try the below command on terminal/command prompt

    C:> assoc .pl
    .pl=Perl_program_file

    C:> ftype .pl
    Perl_program_file=C:\Strawberry\perl\bin\perl.exe "%1" %*

  • if ftype doesn't work we can set it:

    C:> ftype=C:\Strawberry\perl\bin\perl.exe "%1" %*
    

  • If you have installed strawberry Perl you might need to copy libexpat-1__.dll file :

    • From: C:\Strawberry\c\bin\libexpat-1__.dll to C:\Strawberry\perl\vendor\lib\auto\XML\Parser\Expat\libexpat-1__.dll

• Check the registry:

  • Computer\HKEY_CLASSES_ROOT\Perl_program_file\shell\Execute Perl Program\command
    • REG_SZ = C:\Strawberry\perl\bin\perl.exe "%1" %*

• Environment Variables:

  • Make sure the following paths are present in PATH/Path variable:
    • C:\Strawberry\perl\bin
    • C:\Strawberry\perl\site\bin
    • C:\Strawberry\c\bin

• Open ..\pyrouge\Rouge155.py (or wherever you installed pyrouge), go to the function def evaluate(self, system_id=1, rouge_args=None) (it's at line 322at the time I am writing this), and add

  • command.insert(0, 'perl ') right before self.log.info("Running ROUGE with command {}".format(" ".join(command))).
  • Alternatively it can be changed to check_output(command, env=env, shell=True).decode("UTF-8"). (If you don't do it, you'll get OSError: [WinError 193] %1 is not a valid Win32 application.

Installations and Modifications:

• [Optional] Mark evaluation & summ_eval folders as the source directories (those who are using pycharm).
• install pyrouge manually: pip install pyrouge, default version 0.1.3
• Modify the evaluation/summ_eval/rouge_metric.py file:
  • UpdateRouge155(rouge_dir=rouge_dir, rouge_args=rouge_args, log_level=log_level) to Rouge155(rouge_dir=rouge_dir, rouge_args=rouge_args) since log_level is not needed here and causes error.
• Add this to environment path for running file (if using pycharm)
  • ROUGE_HOME=..path\to\your\project\SummEval\evaluation\summ_eval\ROUGE-1.5.5/

Helpful Links

• Perl Script issue: location of ROUGE binary bheinzerling/pyrouge#18
• Perl command options description can be found here: https://github.com/andersjo/pyrouge/blob/master/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl
• Pyrouge Github: https://github.com/bheinzerling/pyrouge
• Setup pyrouge correctly: https://stackoverflow.com/questions/47045436/how-to-install-the-python-package-pyrouge-on-microsoft-windows/47045437#47045437