Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update speed benchmarks #8

Open
annaveronika opened this issue May 13, 2020 · 2 comments
Open

Update speed benchmarks #8

annaveronika opened this issue May 13, 2020 · 2 comments

Comments

@annaveronika
Copy link

annaveronika commented May 13, 2020

We are happy to contribute here.

We suggest following updates:

  • using newest versions of the libraries
  • using 1000 iterations instead of 500, because when you are using 500 iterations preprocessing might be a bottleneck, which is not what you want to measure. Plus you are usually using GPU-s for large datasets where it's not enough to run for 500
  • using two different aws configurations, one with 8 V100 another without GPU-s for running on CPU. It is cheaper this way, and you don't need to pay for GPU-s when you are not using them
  • run 5 times every train on CPU, because for all the libraries CPU time might differ by up to 30% from run to run. So the benchmark will contain average time and standard deviation for CPU

Are you OK with these changes?

@RAMitchell
Copy link
Collaborator

Seems reasonable for me. If we run 5 times, let's do it for all algorithms and have a configurable parameter.

@annaveronika
Copy link
Author

Thanks a lot, it sounds great, we'll make a pr soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants