preprocessing issues

[michaelmoju]

I preprocess the ACE2005 corpus through your code but found some issues. The first issue is that in ace2json.py line93 the print function is in python2. I manually change the print function and run ace2json.py file. But it shows that I miss some files. I found that in ace2005/text/CNN_IP_20030408.1600.04.txt.conll the Standford annotator would wrongly add one more period in line 87~88 the word "SR..". there are two periods. Can you check this issue? Thanks for your contribution.

[luanyi]

Hi Michael,

For preprocessing, we follow exactly the same as Miwa & Bansal's repo . The ace2json.py only converts their output to jason format. I'm not exactly sure what their pre-processing script does for the ``SR." issue. But this code should be fine for direct comparison with previous literature on ACE.

For the missing files, could you give me an example which file you are missing and what error message did you get?

Best,
Yi

[michaelmoju]

In the file: preprocessing/ace2005/log, line165, There is file not found error:

Traceback (most recent call last): File "../common/standoff.py", line 12, in <module> for line in open(sys.argv[1]): FileNotFoundError: [Errno 2] No such file or directory: 'corpus/CNN_ENG_20030304_173120.16.txt' corpus/CNN_ENG_20030305_170125.1.split.txt corpus/CNN_ENG_20030306_070606.18.split.txt

That is because in preprocessing/common/standoff.py line85 asserts whether the texts are the same. However, I found that stanford annotator wrongly annotated the file ace2005/text/CNN_IP_20030408.1600.04.txt.conll line87-88 for some reason which caused the assertion error, and the procedure stops yeilding the following files after the error.