Skip to content

Commit

Permalink
add index_bits option to support large datasets
Browse files Browse the repository at this point in the history
- add `index_bits` optional argument to KDTree
  - default value is 32: preserves existing behaviour & performance
  - user can specify 64 instead to use 64-bit integers
    - this ensures correct results when n_points * k > 2^32
    - uses approx 50% more RAM than 32 bit option
    - resolves storpipfugl#38
- in 32-bit int mode add checks to avoid returning incorrect results
  - KDTree checks that `n_points < 2^32`
  - KDTree.query checks that `n_points * k < 2^32`
- update tests
  - parametrize all existing tests to test 32-bit and 64-bit int mode
  - add a query test with n_points * k too large
  - didn't add a test with n_points too large as it would require 16GB RAM to run
  • Loading branch information
lkeegan committed Jan 17, 2025
1 parent dc38a63 commit b8e1ab8
Show file tree
Hide file tree
Showing 4 changed files with 1,949 additions and 568 deletions.
Loading

0 comments on commit b8e1ab8

Please sign in to comment.