Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Low AUROC #34

Open
spacecraft1013 opened this issue Jul 16, 2020 · 6 comments
Open

Low AUROC #34

spacecraft1013 opened this issue Jul 16, 2020 · 6 comments

Comments

@spacecraft1013
Copy link

spacecraft1013 commented Jul 16, 2020

The description describes a mean AUROC of 0.847, however when I run the model I get a mean AUROC of 0.489, here's the output from running the model.py script:

ambouk3@u109023:/data/ambouk3/CheXNet$ python3 model.py
=> no checkpoint found
model.py:77: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  input_var = torch.autograd.Variable(inp.view(-1, c, h, w).cuda(), volatile=True)
The average AUROC is 0.489
The AUROC of Atelectasis is 0.5147218977086381
The AUROC of Cardiomegaly is 0.4816967965324666
The AUROC of Effusion is 0.5525329228378363
The AUROC of Infiltration is 0.4785231304193095
The AUROC of Mass is 0.5108256156533197
The AUROC of Nodule is 0.4956833510675879
The AUROC of Pneumonia is 0.4754862089500211
The AUROC of Pneumothorax is 0.5329778292671846
The AUROC of Consolidation is 0.49934268439528523
The AUROC of Edema is 0.429807208063108
The AUROC of Emphysema is 0.3639704709500116
The AUROC of Fibrosis is 0.5687886732196019
The AUROC of Pleural_Thickening is 0.443773228540649
The AUROC of Hernia is 0.5042597897539615

I am using the latest version of PyTorch and running it on a Quadro RTX 6000 with a batch size of 8, any ideas as to why this is happening?

@alaadewer
Copy link

i have the same low AUROC, do you have any idea how to raise it?

@bzhangj13zzz
Copy link

Yeah I got exactly the same problem, I wonder if the provided pth file is the trained weights or not

@Aliktk
Copy link

Aliktk commented May 31, 2021

@bzhangj13zzz did you improved the AUROC?

@bzhangj13zzz
Copy link

@bzhangj13zzz did you improved the AUROC?

Hey sorry for the late reply, I did manage to get a more reasonable score, turned out there was some issue with weight loading, anyways I recommend checking out this repo https://github.com/mlmed/torchxrayvision It has a more easy-to-use API.

@alaadewer
Copy link

@bzhangj13zzz did you improved the AUROC?

Hey sorry for the late reply, I did manage to get a more reasonable score, turned out there was some issue with weight loading, anyways I recommend checking out this repo https://github.com/mlmed/torchxrayvision It has a more easy-to-use API.

Please, I want to know how to run this repo using resnet50?

@han-yeol
Copy link

han-yeol commented Oct 4, 2022

In model.py, please rewrite code

if os.path.isfile(CKPT_PATH):
    print("=> loading checkpoint")
    modelCheckpoint = torch.load(CKPT_PATH)['state_dict']
    for k in list(modelCheckpoint.keys()):
        index = k.rindex('.')
        if (k[index - 1] == '1' or k[index - 1] == '2'):
            modelCheckpoint[k[:index - 2] + k[index - 1:]] = modelCheckpoint[k]
            del modelCheckpoint[k]
    model.load_state_dict(modelCheckpoint)
    print("=> loaded checkpoint")
else:
    print("=> no checkpoint found")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants