Reproducing the zero-shot classifier on CXR-14

[sheethalb]

Hello,

thank you for your excellent work. I am trying to reproduce the results of your paper using the CXR14 test set provided by you.
When I run the provided code as is , along with the checkpoint_final.pth , this is the result i get

Load model from checkpoint: /pct_wbo2/home/sheethal/thesis/chexzero/MedKLIP/checkpoints/checkpoint_final.pth
Start testing
The average AUROC is 0.7233
The AUROC of atelectasis is 0.6734893770475145
The AUROC of cardiomegaly is 0.8387400372770306
The AUROC of effusion is 0.8139384383779037
The AUROC of infiltrate is 0.7074241838315987
The AUROC of mass is 0.7431213258915926
The AUROC of nodule is 0.6309411062388708
The AUROC of pneumonia is 0.7065964251060693
The AUROC of pneumothorax is 0.8355626059187432
The AUROC of consolidation is 0.7228660795136705
The AUROC of edema is 0.8016347237756991
The AUROC of emphysema is 0.7614160545496249
The AUROC of tail_abnorm_obs is 0.5860762437031952
The AUROC of thicken is 0.5322912392258987
The AUROC of hernia is 0.7724658820526378
The average f1 is 0.2541
The average ACC is 0.7945

This is lower than the results reported in the paper. Is it perhaps due to a different checkpoint used ? please could you share the checkpoint used to produce the results of the paper? I would like to test its performance on an internal dataset.