Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with reproducing the results #2

Open
JacekMaksymiuk opened this issue Dec 17, 2024 · 1 comment
Open

Problem with reproducing the results #2

JacekMaksymiuk opened this issue Dec 17, 2024 · 1 comment

Comments

@JacekMaksymiuk
Copy link

Hi, I wanted to reproduce your results, but I was unsuccessful.
I wanted to reproduce the results from your paper, but for the football dataset and the Depth-Anything-V2-Large model I get completely different results.
My results:
abs_rel: 0.6578764319419861
rmse: 0.3091728091239929
rmse_log: 1.8111008405685425
silog: 79.3974380493164
sq_rel: 0.2289116233587265

I also tried a bit of postprocessing the data (which is in float32 format), i.e. normalising it on the interval 0-65535, so as to lose as little information as possible, but this helped slightly:
abs_rel: 0.5723931789398193
rmse: 0.25019174814224243
rmse_log: 1.7408266067504883
silog: 71.5643539428711
sq_rel: 0.18551835417747498

Should I apply any more postprocessing? I know that scaling and shifting is done during evaluation, but as we can see this is not enough. How can I reproduce the results from your paper?

Thank you and best regards :)

@JacekMaksymiuk
Copy link
Author

After applying simple transformations (normalizing the result between min(gt) and max(gt)), I was able to go down for the silog statistic from over 70 to about 1.95, but this is still a far from the paper's results.
Have you applied any other transformations on the result besides those implemented in evaluate.py?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant