Quick question regarding accuracy

[RodGal-2020]

Hello there,

After manually executing the training with the AUTSL dataset I got only 0.0045 of accuracy with the VTN-PF model, instead of the 0.9292 achieved with the pretrained model from the releases' section. I re-read your paper, and all the parameters within the code seemed ready for training, is that so or am I forgetting to tune a certain parameter? I have checked OpenPose too, and it looks like it's correctly working.

Thanks beforehand.

[m-decoster]

Hello,

Since there are 226 classes in the dataset and labels are uniformly distributed, 0.0045 corresponds to random accuracy (1/226). This suggests that the labels are not correctly assigned to the samples (but instead are random).

I will have to check this locally once I find the time, but perhaps this can already point you in the direction of a solution.

[RodGal-2020]

I haven't thought about it in that way. I'll write if anything comes up. Thanks!

[youloseiwin]

@RodGal-2020 May I ask if you have successfully reproduced it in the end? Achieve the accuracy stated by the author

[RodGal-2020]

@youloseiwin I believe I finally modified the code and could improve the prediction, but not so much as to reach the expected precision. My contract ended almost a year ago, so I can't remember how It turned up.

[m-decoster]

@youloseiwin Are you having any specific issues? I haven't used this code base in a while but I have recently used the model in a different setting and it works for me. So if you are getting certain errors or random accuracy, there may be some latent issue with this code base.