cannot reproduce the performance of visual Entailment dataset.

[youngfly11]

Hi;
I conduct the pretraining with resent18+3 layer transformer by using indomain data. (without MVM loss)

I can get a similar result on VQA downstream tasks, around 66.5 accuracy.
But the performance on visual entailments is relatively lower than reported in the paper, I can just get 74 accuracy (~82% reported in paper)
I am wondering why the resnet18+3 layer outperforms the Uniter Base?
Are there any training strategies specialized for this downstream task?

Thanks

[youngfly11]

hi, again
Is there any plan to release the Visual Entailment codes or process datasets? I still cannot reproduce the performance. Everything goes well except the VE tasks. It is very strange
Thanks

[alice-cool]

Hi; I conduct the pretraining with resent18+3 layer transformer by using indomain data. (without MVM loss)

I can get a similar result on VQA downstream tasks, around 66.5 accuracy. But the performance on visual entailments is relatively lower than reported in the paper, I can just get 74 accuracy (~82% reported in paper) I am wondering why the resnet18+3 layer outperforms the Uniter Base? Are there any training strategies specialized for this downstream task?

Thanks

Dear scholar, the picture warining is normal or I missed some files?

[mactavish91]

hi, again Is there any plan to release the Visual Entailment codes or process datasets? I still cannot reproduce the performance. Everything goes well except the VE tasks. It is very strange Thanks

@youngfly11 I think they use both image premise and text premise, causing information leakage and resulting in performance far exceeding other models.

[Gavin001201]

hi, again Is there any plan to release the Visual Entailment codes or process datasets? I still cannot reproduce the performance. Everything goes well except the VE tasks. It is very strange Thanks

Hello, I want to know if you finally figured out the reason and whether there is a problem with the implementation of VD here.Thanks