Error during eval - can't find tokenizer

[salfamusic]

Hello,

I am getting the follower error when trying to run the eval:

Traceback (most recent call last):
  File "./eval.py", line 2, in <module>
    from tokenizer import SimpleTokenizer, PretrainedTokenizer
ModuleNotFoundError: No module named 'tokenizer'

I have searched the repo and cannot find a module named tokenizer. So I imagine this must come from one of the dependencies (probably the transformers dep?).

Can you help out? Thanks

[1245244103]

Hello,

I am getting the follower error when trying to run the eval:

Traceback (most recent call last):
  File "./eval.py", line 2, in <module>
    from tokenizer import SimpleTokenizer, PretrainedTokenizer
ModuleNotFoundError: No module named 'tokenizer'

I have searched the repo and cannot find a module named tokenizer. So I imagine this must come from one of the dependencies (probably the transformers dep?).

Can you help out? Thanks

I have the same problem.Do you solve it?

[zhenzliu]

hello , i have another problem,when i bash finetune.sh, finetune.py not run (i add some print orders in finetune.py,but no print).
are there some issues that i neglect？
thanks for your help!

[1245244103]

hello , i have another problem,when i bash finetune.sh, finetune.py not run (i add some print orders in finetune.py,but no print).
are there some issues that i neglect？
thanks for your help!

你是跑完fineune.py没反应？

[zhenzliu]

hello , i have another problem,when i bash finetune.sh, finetune.py not run (i add some print orders in finetune.py,but no print).
are there some issues that i neglect？
thanks for your help!

你是跑完fineune.py没反应？

是的，finetune.py没有运行起来

[1245244103]

hello , i have another problem,when i bash finetune.sh, finetune.py not run (i add some print orders in finetune.py,but no print).
are there some issues that i neglect？
thanks for your help!

你是跑完fineune.py没反应？

是的，finetune.py没有运行起来

你huggingface的bart_model下载下来了吗？不然的话，那个命令的model_name_path得改一下

[zhenzliu]

我下载下来了

[1245244103]

我下载下来了
唔，你试着挂一下vpn？我好像遇到过几次没反应的情况，但我终端走代理之后，就没遇到过了

[Richar-Du]

Hello,

I am getting the follower error when trying to run the eval:

Traceback (most recent call last):
  File "./eval.py", line 2, in <module>
    from tokenizer import SimpleTokenizer, PretrainedTokenizer
ModuleNotFoundError: No module named 'tokenizer'

I have searched the repo and cannot find a module named tokenizer. So I imagine this must come from one of the dependencies (probably the transformers dep?).

Can you help out? Thanks

Replace it with the following code:

from utils import SimpleTokenizer

Because the SimpleTokenizer is defined in the eval/utils.py, and the PretrainedTokenizer isn't called.

[LiuTianSh]

hello , i have another problem,when i bash finetune.sh, finetune.py not run (i add some print orders in finetune.py,but no print).
are there some issues that i neglect？
thanks for your help!

你是跑完fineune.py没反应？

你好，请问您知道最后的最优模型是存在best_tfmr下面吗？直接按照给的方式无法跑generation。