Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

train is ok,but evaluate OOM #3

Open
TheHonestBob opened this issue Mar 16, 2022 · 1 comment
Open

train is ok,but evaluate OOM #3

TheHonestBob opened this issue Mar 16, 2022 · 1 comment

Comments

@TheHonestBob
Copy link

thanks for your project, when I sh finetune.sh,OOM happen when evaluate,even though I set eval_batch_size=1,my gpu is 2080ti 11G.

@ZhuohanX
Copy link

Hi,
Could you please help me to indicate the versions of packages you are using to run the code?
I tried Python 3.7.0 with torch==1.4.0 but it seems that module 'torch.cuda' has no attribtue 'amp' because I think it is included in torch 1.6.
And then I changed my package version to Python 3.8 with torch 1.7 but I got an error when running the fine-tune.

File "/home/zhuohanx/HINT/model/utils.py", line 294, in collate_fn
batch_score.append([float(s.split()[2]) for s in score_list])
File "/home/zhuohanx/HINT/model/utils.py", line 294, in
batch_score.append([float(s.split()[2]) for s in score_list])
IndexError: list index out of range

I am not sure if it is caused by the different version or an error in the code?

Thank you in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants