Cannot reproduce experimental results on Openlane?

[inconnu11]

Hello all,

I ran the Openlane experiment based on the provided codebase and didn't change any parameters in the config file. I tested multiple models saved at different epochs and only got the best results of 56.3% F-score on Openlane validation set, which is still 2% lower than the number shown in your paper (i.e., 58.4% F-score). Is there anything wrong? Could you please give me some advice?

[qinjian623]

hi,

You can try adjusting the training hyperparameters, as different numbers of GPUs and hardware environments may yield different results.

It is recommended to adjust the learning rate (lr) and batch size.

Using this version of the open-source code, we can achieve F1 score of 60.

[yuaoze]

hi,

You can try adjusting the training hyperparameters, as different numbers of GPUs and hardware environments may yield different results.

It is recommended to adjust the learning rate (lr) and batch size.

Using this version of the open-source code, we can achieve F1 score of 60.

hi,
I've tried and get the best result of F-score is 57.4% on Openlane, My device is 2x3090, batch_size is 32, other hyperparameters are same as the code. It shows that loss oscillated around 5 without decline. Could you please give me some advice?

[secret104278]

HI, @inconnu11 and @yuaoze, is there any possible that you could share your own trained model for quick evaluation?

[qinjian623]

hi, @yuaoze

We use a hardware configuration of 8 x V100, which you can use as a reference to adjust hyperparameters.

[yuaoze]

sure, please leave your email

[Laulian]

sure, please leave your email
@yuaoze
[email protected] ~ It will be greatly appreciated

[secret104278]

sure, please leave your email

[email protected]

thanks a lot ~

[zhanghui75]

Could you also kindly share your trained model with me? Thanks a lot.
My email: [email protected]

[imyyf]

hi,
You can try adjusting the training hyperparameters, as different numbers of GPUs and hardware environments may yield different results.
It is recommended to adjust the learning rate (lr) and batch size.
Using this version of the open-source code, we can achieve F1 score of 60.

hi, I've tried and get the best result of F-score is 57.4% on Openlane, My device is 2x3090, batch_size is 32, other hyperparameters are same as the code. It shows that loss oscillated around 5 without decline. Could you please give me some advice?

Hi
which epoch you choose to get 57.4% on openlane? we use epoch049(batch_size is 32) to test, only get 53%.

[adeelahmedrana]

Hi, first of a very impressive paper.
Can you please share the weights with me too at [email protected]

Many thanks

[Yutong-gannis]

@yuaoze
Could you please share your weights with me just for a early demo. Thanks a lot.

My email: [email protected]
[email protected]

[westlife35]

@yuaoze
Could you please share your weights(57.4% on openlane). Thanks~
My email: [email protected]

[superchenyan]

sure, please leave your email

my email: [email protected] ths!!!

[drilistbox]

could you share your weight to me, my email is [email protected]

[PyNancy]

sure, please leave your email

could you please share your weight to me, thanks a lot, [email protected]

[WwwwYz666]

Could you please share your checkpoint weight ? My email is [email protected].

[CaiFuyao]

could you please share your best model to me , thanks a lot! My email is [email protected].

[s95huang]

Here is what I got,
{'f1_score': 0.5651117116621163,
'precision': 0.6838964779914308,
'recall': 0.4814843714408606,
'x_error_close': 0.25393218394610845,
'x_error_far': 0.6785732514552418,
'z_error_close': 0.19615825120228755,
'z_error_far': 0.6227202588181409}

[Pocan0]

Here is what I got, {'f1_score': 0.5651117116621163, 'precision': 0.6838964779914308, 'recall': 0.4814843714408606, 'x_error_close': 0.25393218394610845, 'x_error_far': 0.6785732514552418, 'z_error_close': 0.19615825120228755, 'z_error_far': 0.6227202588181409}

could you please share your model to me , thanks a lot! My email is [email protected].

[Mollylulu]

@yuaoze Could you please share your reproduced model weights with me? Thanks a lot!
Email address: [email protected].

[EnternalTwinkle]

@yuaoze Can you share the source code? Thank you very much，[email protected]

[Ywenjun123]

Hello all,

I ran the Openlane experiment based on the provided codebase and didn't change any parameters in the config file. I tested multiple models saved at different epochs and only got the best results of 56.3% F-score on Openlane validation set, which is still 2% lower than the number shown in your paper (i.e., 58.4% F-score). Is there anything wrong? Could you please give me some advice?

could you please share your model to me , thanks a lot! My email is [email protected]

[happyday-lkj]

Here is what I got, {'f1_score': 0.5651117116621163, 'precision': 0.6838964779914308, 'recall': 0.4814843714408606, 'x_error_close': 0.25393218394610845, 'x_error_far': 0.6785732514552418, 'z_error_close': 0.19615825120228755, 'z_error_far': 0.6227202588181409}

could you share your pretrained model, [email protected]

[spring35]

sure, please leave your email

could you share your model, thanks! my email is [email protected]

[Benson722]

Could you also kindly share your trained model with me? Thanks a lot.
My email: [email protected]
thx!